Overkill's devlog

Overkill

There was talk of starting up “Gruedorf” again – a little competition made by McGrue and Kildorf, where all entrants would be required to work on something each week and post something about it. Once a post is made, you have 7 days (168 hours) to make your next post, or you “lose”. The goal is to keep a consistent stream of progress each week, and try to minimize “losing” by doing a little work and posting about it.

As far as I know, there is no requirement that the project remain the same throughout all posts, which I’m thankful for. I accept that at this age that my attention span is all over, and I constantly cycle projects even on the best days. Although, finishing something is better than meandering between 3000 unfinished things that get abandoned for the next thing… Probably other rules I’m forgetting.

Anyway, this sounds like fun. Like many regular routines in my chaotic day-to-day life, and like previous attempts when I had way more free time, I’m not sure I’ll be able to stick to any sort of consistent schedule, but going to give this a try in any case!

With full-time work, and general lack of time organization, I have a few projects going on that I am cycling between depending on my general energy. Here are the ones I can remember off the top of my head, but it is not an exhaustive list:

Wiz, a high-level assembler. The compiler is a work-in-progress, but it’s already capable enough to write demos and games (and comes with working example programs included). It’s a strongly typed language, with C-like structured programming that compiles directly to machine instructions on different retro CPU systems (6502, 65c02, HuC620, Z80, GB, 65816, SPC700). It has gathered a small community on its Discord channel (linked in the project page). This helped stress-test the language, and uncover neat uses I didn’t expect were possible with the language, as well plenty of hair-pulling bugs. But it’s still nice that this tool has a userbase and came in useful to others, even if it was written as a hobby project for my personal use at first. Hopefully more iterations can be made to improve the syntax, the documentation, the compiler tooling, write more example programs and games, add more platform backends, etc.

Dive into the “examples” folder included with the repo to see some examples of things you can do with it. I’ve included a Hello World for many systems, as well as a partial shmup demo for NES, a virtual pet sim on GB, a multiplayer VS game with ducks on Atari 2600, a few VFX tech demos, some libraries for some hardware register definitions and common utilities (GB and NES are most right now, but others are grown as examples and personal projects are made), and some other miscellaneous things.
nrpg, a turn-based RPG for the NES. This is written with Wiz, and is a real NES program. I’ve been wanting to design a game that has tile-based overworld puzzles such as Final Fantasy Mystic Quest, Lufia 2, Fairune, and others. It will have a battle system similar to classic FF games, with maybe a sprinkle of the SaGa series.

So far I have a working MMC3 tech demo ROM that allows the player to walk around an top-down map with tile-based map. The map can scroll in all four directions without visible scrolling artifacts. The player is so far able to move across the map and jump (like FFMQ and FFL3/SaGa 3), but there is no support for interactive tile objects, zones or NPC system yet. There is also menu system with a working inventory subsystem. Still early, but the foundation needed to make an RPG is slowly coming together. I had to shelve it for a bit but I want to pick it up again soon.

Here’s some screens and mockups (note some GIFs needed to crunched to 1x size to fit in imgur upload limits)

(tileset / map mockup)

(item system - capture from the NES ROM)

(overworld jumping mechanics - capture from the NES ROM)

(a mockup of a battle scene – not yet implemented in game, in typical Verge community style of not having a battle system yet)
Wandering Magic, an action RPG, which started as a PICO-8 demo title (released for the 3-color Jam in late 2017).

Here is the original jam entry – you can play in your browser, and it’s about 30 minutes in length. It features simplified Ys-like bump mechanics for the combat, as well as simple leveling system that balances fast-paced grinding with finding/buying new items to enhance your combat abilities. Most areas are gated by your player’s stats (health, attack, defense), and equipment works as key items.

I wanted to rewrite the game, so I ported the game to regular Lua + Love2D, as a cleanup exercise to make the PICO-8 source legible code again. But eventually, I wanted to rewrite the game again in C, as something that can compile out of the box with SDL, doesn’t require a dynamically-typed scripting language, and doesn’t depend on a heavyweight engine or content pipeline. I also wanted to double the game’s internal framerate to 60FPS (rather than PICO-8’s default 30FPS), increase the size of art assets (already redrawn but not integrated), write a coherent quest, and make the action more compelling. The bump combat mechanics were neat, but sometimes it felt like the enemy melee damage could be telegraphed better so the player could evade things more easily.

(dark mages teleport around and shoot fireballs at you, as slimes spawn in from the sides.)

(fighting a skeleton that guards the entrance to the next area.)
vg - a virtual console system in C (working name, but might stick). This probably requires some elaboration – it simulates the experience of writing games for older consoles, but picks limitations that can be nicely fitted to a modern widescreen 16:9 desktop with gamepad support.

It calls two user provided entry points, one for resetting the game, and one for updating the next frame. All data shared with the engine is global arrays, or through functions that act as ‘system calls’ that convert user-provided data into backend internal texture formats/sound formats/etc. Games are able to tileset graphics and palettes, and then draw them using either tilemap or sprite primitives, which are hardware accelerated. The user of the engine doesn’t need to know the nitty gritty about how the graphics get internally represented (it may be different depending on the platform), and can talk about things in terms of 2D tiles and sprites. Ideally, all games include all data directly as static global data in the source program, and work without solely using the exposed engine functionality. Programs can be written using no stdlib/stdio (except internally to the engine), so to allow extra portability, minimal load times, avoiding the runtime overhead of dynamic memory management by using the static data segment instead, and minimizing runtime issues relating to file I/O.

The screen is resolution is 320x180 (very close to the 320x200 used in the Verge days, but it’s 16:9). Backgrounds can be made from 3 tilemaps of 64x64 each, which can either be comprised of 8x8 or 16x16 tiles. You also get 128 sprites which can be between any of the 3 layers. Tiles and sprites can have 16-step alpha blending applied, and palette remapping for hitflash effects and color swaps. Tilesets are capable of holding up to 512 16x16 tiles at once. The sprite layer and individual tilemaps can select which 256-tile window they are allowed to see of tileset (so one layer could point at a font, while another could be used for the main background graphics or sprites, the arrangement is up to you), Palettes are 256-color 24bpp RGB data. It’s possible to update the tileset graphics on the fly (to load another tileset or modify the current one), and upload new palettes. Input supports gamepads or keyboard input. Persistent saves of up to 16KB are possible (this limit might be expanded)

For audio, it supports wavetable sound synthesis (similar to the PC Engine, SCC soundchip on MSX, Namco N163 expansion chip on the Famicom, GB wave channel, etc) with 6 channels. There is storage reserved for holding short 32-step wave samples and instrument envelopes. There is an MML-like SFX command data that can play different notes, change tracker speeds, apply pitch slides, note slides, vibrato, etc. Playing songs or sound effects works by playing different pieces of the sfx data memory, and is written so that multiple sfx issued on the same frame are played together (for allowing polyphonic music with synchronized channels). The audio thread can run on its own, in parallel with the main game logic, but receives updates from the main thread at roughly 60FPS during normal circumstances.

There’s tools for importing tileset assets from PNG files, as well as converting text files containing an MML-like domain specific language into binary blobs that contain SFX commands.

So far I’ve managed to get a working engine with an example program that I wrote at the same as the engine, need to write actual game with this next. There is a backend currently for SDL that so far compiles on Windows, Raspberry Pi 400, and Emscripten with minimal dependencies (user-installed SDL on non-Windows). It is intended to be easy enough to write other backends in the future, so that porting to non-desktop platforms is a possible option.

Wandering Magic might be a good candidate to port to this engine, since it has enough features at this point.

(a little demo mockup)

…Wow that was a lot more rambly than I intended, lol. This initial post is a quite wordy summary of some projects I have going on and the state they’re in, but intending for future posts be more short and concise, unless something interesting warrants more elaboration. Let’s see if the habit of posting here will stick! Hopefully other folks will be posting on here too, to keep things interesting.

Overkill

Unfortunately, this week I didn’t get as much new work done as I had hoped, because my time was pretty divided.

However, I did manage to contribute some changes to the v2wasm project, which is a project started by Andy Friesen / thespeedbump to emulate playback of Verge 2 games on the web. I made a few tweaks to improve its compatibility on certain verge2 games. These can hopefully be merged upstream soon.

Added more key[] bindings work in v2 games, by mapping more virtual key code into DOS/Win32-style physical key order that VC’s key[] array expect.

Note that while this solution works on desktop devices, it may have issues with non-QWERTY physical key layouts.

It also doesn’t address key[] usage from a mobile/tablet device which has no external keyboard – something that v2wasm will want to address later. Possibly we can handle this entirely in the web player that comes with Verge Archive, .
Added a compatibility option in user.cfg named recursive_render, that enables Render() to be called recursively with from a Hook Retrace callback. v2wasm is based on Verge 2.6 which disables recursive rendering by simply returning if you Render() while one was in-progress.

However, this particular decision in 2.6 causes an compatibility issue with Beyond Babel, a game written for Verge 2.5 and winv2. It relies heavily on manual render loops that can be called inside of hook-retrace callbacks. During combat, it handles battle sub-menus by starting different menu loops within its retrace handler. To prevent infinite recursion, it calls HookRetrace(0) and manually Render() + ShowPage(). By unbinding the retrace function, it will do nothing when hitting the R layer while already in a retrace, and then restores the retrace hook after the menu is done.

Since other games might rely on the 2.6 behaviour to not crash, this was made into a user.cfg setting that is off by default. Sort of unfortunate this settings needs to be added manually, but I think it’s the best way to support both versions without requiring VC code changes.

The short version: a new compatibility switch fixes the screen not clearing correctly during battles in Beyond Babel, it might fix other games too.
Fixed the “manifest.txt” file listing for Beyond Babel so it’ll be possible to play in the Verge Archive player.
Added support for Fugue to the Verge Archive player. It seems to work pretty well!

I also looked at Wiz, my high-level assembler, a little bit. I received some new bug reports I want to get around to soon, and I also thought a little bit about how to fix some things. No new commits, but planning is just as important when it comes to projects like this.

using pointer-to-struct to access nested struct members doesn’t work, when indexed addressing modes are available:

struct Stats {
    hp : u8,
}

struct Monster {
    stats : Stats,
}

var monster : *Monster in ix;

monster.stats.hp = a; // error.

This is because currently, the compiler isn’t able to fold the index offset expression into a single constant. Converting the struct member access into indirected addition results in a reduced expression that looks like

*(((ix + offset1) + offset2) as *u8) = a;

to the pattern matcher, which isn’t a valid instruction on the Z80 CPU. However, *((ix + displacement) as *u8) = a exists on Z80! If the compiler could more aggressively fold constants together in this case, Then the issue would go away, and people will be able to use registers as general struct pointers.

Currently using a far function won’t validate that all call expressions are also far. When the calling convention doesn’t match on 65816, this results in runtime bugs. This is because a subroutine called with JSL must return with RTL, and a JSR must return with RTS, so the call and return convention must match.

Wiz should make 1) far funcs imply far calls everywhere, or b) make it an error to call a far func without far. On platforms where far and near calls can be mixed without introducing problems, I might relax this restriction.
I also did a little planning for an eventual refactor I would like to do to Wiz’s C++ codebase. I’ve been trying to break out the compiler code into smaller parts in an incremental way.

The compiler.cpp is a 7000+ line source file, and it is becoming a bit messy in its current state. Since passes of the compiler depend on each other, it’s not a super easy thing to just start splitting stuff up immediately. But there are definitely some subsystems and passes that don’t interact, as well as some common things that can be moved into their own file, with clearer data structures and code to operate on them. With a bit of cleanup, the project will be easier to read and maintain. Things would be more clearly separated, and it could eventually add test seams for unit testing.

(Some candidates for clean-up: the scope stack + name resolution, the attribute stack, the inline stack, expression reduction (constant expression folding + typing + qualifying the expression trees), type-checking, type resolution, instruction operand creation, statement-to-IR compilation, IR passes)

The code should have been written with unit tests in mind earlier on. However, I don’t know if Wiz would have ever made it to as far as it did, if it were encumbered by TDD when the basic language design was still being figured out and implemented.

We do have some functional tests, but these run the compiler as a kind of black box, confirming the compiler will generate certain machine code or produce specific errors. Having more compiler-internal unit testing would improve things a lot. Not only would it aid in ensuring build stability, but it would also serve as documentation for some of the finer details in the compiler.

Going to try to aim for some low-hanging fruit to start moving things out, and hopefully this will make it clear about other subsystems that can be split up next. Then once there are more. I know this order to write tests after-the-fact, especially after a refactor, is a little less conventional. But my thinking is that it probably won’t be possible to write good tests until the code is smaller organized chunks with easy tests to begin with. I need to dig things out of the hole they’re in first.

Still not rushing this, but thinking about it.

Aside from that, I drew a little bit. I made this little pixel sketch;

The background is a little bit too visually noisy, and needs a bit more variety in the wall tiles, but it was still fun to draw something. It could be cool to make the walls more crystalline and glossy/reflective (maybe some bright white speculars), and better hue + value-separate the background and foreground tiles.

Anyway, that’s all for this week!

Overkill

This week is very short to summarize. Other parts of my life have been competing for spare time lately. But in spite of that, I did manage to squeeze in a couple hours of work on Wiz.

I improved integration with debugger integration with Wiz programs. I added some work to improve debug symbol sorting in wiz, so that fully-qualified / global names have precedence over anonymous / compiler-generated names. This means that it will be much easier to search for a particular symbol by name in the debugger and go to it – and its given name (since Mesen annoyingly only allows one alias for each address of memory) will prefer a deliberate user-chosen name over auto-generated stuff.

I have wanted to also fix Wiz’s symbol exporting for the SNES.

The problems are better explained by first explaining what a “debug symbol” is – basically, a label or identifier that gives a name to an address in memory, constant, or similar thing. This allows you to view these symbolic names rather than hex values inside of a debugger’s disassembly viewer. These give names to better indicate where functions, labels, variables, and constant data appear in the program.

The problem is, currently the way addresses are mapped to the symbol file is incorrect. As soon as you have a program outside of bank 0 of ROM, the address calculation is wrong. Or rather, wrong depending on the ROM’s map mode that is used.

The SNES is peculiar because the system allows several different ways to access the same memory areas. Firstly, on the CPU-side, the 65816:

you can access an 8-bit address, which is offset by the direct page register existing somewhere within bank 0 of the memory space.
you can access a value on the stack.
you can access a 16-bit absolute address, which has its upper byte of the effective 24-bit determined through the its data bank and program bank registers.
you can use 24-bit absolute long addressing. It allows you to access the full 24-bit address space of the SNES memory map. This is a similar concept to “far addresses” in 8086 parlance, but bit simpler. For familiarity, Wiz also names this far addressing.
you can index an 8-bit direct page address, 16-bit absolute address, or 24-bit absolute address.
you can use a variable within the direct page or stack as a pointer to a 16-bit or 24-bit address in memory.
Also, there’s two different sizes of stack-based call/return, which use different lengths of addresses:
- 16-bit jumps with JSR (jump to subroutine) and RTS (return from subroutine).
- 24-bit jumps with JSL (jump to subroutine long), and RTL (return from subroutine long).
and so on (there are more address modes)

At the end, the operands to these addressing modes are decoded at some point into an effective address by the CPU. The way this is done is determined by the memory map of the SNES, which is configurable by the cartridge.

There are LoRom, HiRom and FastRom SNES roms. (Going quick to not miss the Gruedorf post deadline hahah. But edited slightly for clarity and typos.)

Here’s a small summary of the “map modes” on the SNES cartridges:

LoRom lets each bank of memory you access view of ROM in 32K blocks and the 32K of address space contains other stuff (RAM, the hardware registers, etc). These are located at banks $00 .. $3F / $80 .. $BF.
HiRom also lets you view your memory like LoRom (in banks $00 .. $3F) – but if you do this, each bank of memory is actually 64K, you only access the upper 32K in each view. But there’s another ranges of bank address that let use this as a full contiguous 64K block (in banks $40 .. $7D, or $C0 .. $FF). This potentially works better with games that have different data banks and program banks, so the program bank can point to a 64K area, but data bank if, or each can point at a different area.
HiRom allows more memory to be accessed at once, but requires using banks that contain only ROM space when they are selected, which means you will need to manage your data and program bank registers more, or use 24-bit far addressing more often. HiRom can still be used in LoRom fashion, but will be less efficient when used this way. It steps in 64K increments, but only sees each the upper 32K of a 64K block when viewed in the LoRom compatibility banks.
LoRom has less memory it can view at once and less maximum capacity, but its modes may be easier to work with because even if the ROM bank changes, the program can still access RAM and at the same time – which might be useful if you want to copy stuff within the same page using faster+smaller 16-bit instructions. It always steps in 32K increments.
FastRom mirrors the entire LoRom/HiRom address space in the upper area of the bank address space, but starts from $80, $81, ... instead of $00, $01, .... This high bit in the address line is used as a signal that fast access mode can be made (I think the cartridge hardware is responsible for checking this after a game software activates “high speed mode” on the SNES, but the exact details don’t matter for writing a program.)
The main takeaway from this is, the same memory area in the ROM can be accessed in many different “mirrored” banks of memory. (there’s more nuances if you get into expansion chips, or weird third-party cartridges which could have their own memory mapping schemes)

Emulators such as bsnes-plus and Mesen have opted to, for whatever reason, import addresses from the symbol file as absolute addresses that are unmirrored (even though the addresses when fetched by the CPU are mirrored) As a result, this means a name can map to exactly one absolute address – no mirroring occurs.

Maybe because that’s what WLA DX already used? Not sure! Anyway, so I need to adapt the debug code to use the absolute bank addresses exactly as they’re declared in the Wiz program, which are normally meant to relocate the program counter during assembling, but aren’t necessarily the true position in the ROM. Before this, Wiz was only using output-relative offsets: This is a better, relocatable concept, which refers to the exact address in the ROM, and the emulator could load this with mirroring in mind to resolve things. However, since not every emulator works like this, and we don’t want to fork existing emulators just to incorporate what we think would be better, a change is needed.

I haven’t had a change to sit down and support this mode yet. I need some way to make these two addressing schemes be selectable by debug target:
a) output-relative byte offset, or
b) cpu memory map address space.

This way, WLA DX-style .sym exports conform to CPU memory maps on all symbols. I’d like to make the default settings for auto-detected by debug symbol file format, to make it so that exporting doesn’t have to configure this. Need to think on this a little more.

Anyway! Hopefully soon!

Overkill

This past week, despite being very busy, I managed to be a little more productive than previous weeks.

I closed a few bugs in Wiz:

Fixed SNES debugger symbols to work again, so now you can get nice named labels and variable names in Mesen-S and bsnes-plus when debugging SNES programs.
Someone reported a Mac OS crash in the compiler! It was caused by C++ argument evaluation order being unspecified. This means that the order depends on the compiler vendor (usually dictated by the calling conventions of the target system).

Usually this is easily avoidable, however std::move is a core feature as of C++11, which will transfer ownership of resources (eg. using unique pointers to enforce single ownership and prevent memory leaks, or for avoiding expensive copies of heap-allocated data structures). For unique pointers such as std::unique_ptr (or Wiz’s equivalent-but-better-inlined wiz::UniquePtr), moving is necessary, and the copy assignment and copy constructors are deleted to enforce move semantics.

There is is a bit of a hidden “foot-gun” waiting to go off though: std::move has the side effect of moving its contents to the destination and nulling out the source afterwards, to preserve the single ownership of a piece of memory. This is how std::move is meant to work, so nothing unexpected here so far. But often, there is the need to move a value AND access some its members in the same call.

Code like f(std::move(ptr), ptr->member); is easy to slip into the codebase by accident, especially as compilers don’t generate any warning for this. On x86-based Windows machines, where the argument evaluation order is right-to-left, this code will compile and run as expected. On Mac OS, this will compile too, but it will always crash at runtime, because it evaluates function arguments left-to-right – this means the std::move(ptr) is called before ptr->member is evalated.

The fix to this crash is to store things into locals before a move occurs, so that you don’t use-after-move. But the nature of this being “unspecified” by the C++ standard means that the compiler never warns about this bug, and it may even appear to work if the vendor/platform-specific evaluation order just happens to works in the program’s favor. As a result, any change to the code could introduce this bug without notice. This reliance on the evaluation order is never desirable when wanting to write portable cross-platform programs in C++ that run reliably (and without crashing).

Thankfully, clang-tidy is a useful static analysis tool that can detect these sorts of problems by running with the bugprone-use-after-move check enabled. It caught the crash and a number of others within the codebase.

VS2019, for all of its bloat, even comes with clang-tidy. This would be great, but MS unfortunately messed this up somehow and shipped an update that breaks being able to do static analysis – I had to manually patch my VS2019 install’s version of intrin.h to get things working. Sigh. Anyway. A long journey later and things are working better.
Fixed the issue with nested struct access, Now you can declare an alias for register, and use as a pointer-to-struct by going var monster : *Monster in ix; and then use it like monster.stat.atk = 10;

With that weight lifted, I started to get planning out the battle system and monster encounters for my NES RPG.

I thought that it might be useful to design the encounters in a sort of “top-down” kind of way. having families/categories/taxonomies of monsters that all share common traits and behaviour. Within each family, the monsters would be divided into tiers, a sort of “evolutionary hierarchy” (for monsters) or “chain of command” (for humanoid types), where each tier is a higher difficulty and more powerful.

Then, monsters families themselves belong to different environments. Things would be designed such that there’s a nice variety of different encounters spread across the environments – gradually increasing in difficulty, or introducing a new challenge or mechanic to the fights to keep things interesting.

My hope is that this sort of approach will make it easier to get a general image of how the game should “look”, and allow me fill in boxes as I go. This should give a more coherent idea of the big picture of how things flow across the game area-to-area, something which usually gets lost when doing in a bottom-up method, one-map-at-time.

Monsters themselves would have distinctive behaviour, and use a combination of some state-machine driven logic, turn-based countdowns, and reactions (counters). Each state would be capable of producing a list of actions that it wants to consider, as well as a preference score to each one. States could transition depending on specific conditions like HP %, being hit X many times, detecting the player doing something, etc. Monsters themselves could also track grudges, prioritize protecting other monsters and other stuff.

But I am thinking of implementing monster AI in a way that it’s mostly driven by the monster’s state and a few other scratch variables. I think this method of monster design could be pretty versatile despite being fairly simplistic.

I also had some thoughts about the battle system itself. I had originally made a mockup in FF6 style, but I was thinking it might be nice to use something a little different from ATB battles. I was thinking it could use initiative-based turn orders with a combination of randomness + character speed to help determine the order. Combatants would decide and act at the same time when their turn comes up. Some actions could potentially have a charge time (that resolves as its own turn between combatant turns). Actions could also have a cooldown that affects the timing of a combatant’s next turn.

I also drew this little mockup of Wandering Magic if it were a game for the MSX 1:

The mockup tries to follow MSX “Screen Mode 2” restrictions – 256x192 resolution, a fixed 16 color palette, backgrounds are composed of 8x8 tile patterns that can have 2 possible colors per 8x1 area, sprites are 1-color and 16x16 in size.

That’s all for this week!

Overkill

This week I ended up expanding on that MSX mockup of Wandering Magic, and started planning a possible true MSX port of the game.

Then I sat down and learned a bunch about coding for the MSX1.

Along the way, I learned the MSX has a pretty extensive BIOS that provides many routines for common interactions with the hardware, so it took some learning to do things the “best practice” way. Some performance might be sacrificed, but it ensures compatibility between versions. And there are also faster methods for looking up I/O ports for some things, for stuff like graphics coding – where it probably matters most.

I updated Wiz’s MSX “hello world” example to use these BIOS routines: https://github.com/wiz-lang/wiz/blob/master/examples/msx/hello/hello.wiz

Previously I thought you had to write to I/O ports directly if you wanted any sort of speed, since I’m so used to that with other systems, and figured the BIOS was for BASIC programs only. But happy to be wrong here, and it’s easier to write code this way too.

Now back to Wandering Magic. I also coded some sprite drawing code that handles scrolling the sprites off both sides of the screen, and an entity system (so far just spawning/updating/drawing – no collisions, no gameplay specifics yet). I also split the art work into bg and sprite tiles, and convert them using tools.

Next up, I want to draw the background map, and see if I can get some player input and movement. The MSX has tons of RAM available, so it’s probably possible to unpack the entire room into individual 8x8 tiles in RAM, and transfer it to the MSX’s tilemap VRAM afterwards. I guess I’ll try this soon!

See you next week!

Overkill

Hello! This week, I made more progress on the MSX port of Wandering Magic in my spare time in the evenings.

I managed to implement a few things that are pictured in this GIF.

First off, I made a map loader, It loads an uncompressed tilemap from ROM into RAM, each part of the map is made up of 16x16px blocks called “metatiles” – 2x2 arrangements of 8x8px hardware tiles, along with some associated metadata like collision flags.

In addition to loading the tilemap into RAM, the metatiles for each cell are then unpacked into the raw 8x8 tilemap format that the MSX VDP expects, and this will be copied to the screen between room switches.

The upper area of the screen that contains playfield of the game is 16 tiles wide x 10 tiles tall. The lower part is reserved for HUD elements. The camera doesn’t scroll – the MSX1 is bad at hardware scrolling, and I designed the game around having individual screen-sized rooms.

Next, I added player input and movement with the keyboard. That was pretty fast! One thing that was a little tricky is that the MSX returns a directional code from 0 - 8 (0 = no direction, 1 = up, 2 = up-right, 3 = right, 4 = down-right, etc. going around the circle clockwise), rather than having a separate input for each direction. This is at least if you use the standard “direction reading” routine from the MSX BIOS, so turning that into independent x and y axis takes a little bit of translation. But a nice feature of the BIOS, is that it standardizes reading both keyboard or joystick for directions and buttons.

There’s also a way to read the keyboard matrix directly, which is necessary if you want to use other keyboard keys besides arrows + space bar. It seems pretty common to find MSX games that use arrow keys, space bar as a primary action, M as subaction, function keys for menus, and support 2 button gamepads. So I guess eventually I’ll need to look into all of that.

After that, I added entity movement with tile collision support. The entity movement was pretty challenging to write but after some trial and error and lots of debugging to step through things, I managed to make something functional,

The collision code even supports a neat feature – corner nudging! This will automatically slide the player around corners when it detects only one side of the collision test is obstructed. For example, if you’re walking up into a corner that is obstructed to the right but not the left, then the movement code will redirect your movement to the left, off the obstruction. As soon as the way is clear again, the movement code goes back to behaving as normal, proceeding in the direction the player inputted. Corner nudging makes it much nicer to do pixel-based movement through narrow spaces (especially single-tile wide ones), without needing to lessen the size of the player’s collision box too much.

Collision still needs a lot of work to optimize things better, since it does slow down when there are several entities on screen. For testing this, I duplicated the player entity 32 times. I observed the framerate dip as soon as the movement began, with even more slowdown happening when moving on diagonals (since it needs to test on x and on y). When the entities are still, or there are less entities, no slowdown occurs, This slowdown problem also occurs at 16 of 32 entities, but only when moving in both directions at once. I’m not sure the exact tipping point where things spill out of the available frame time, but there’s a bunch of things that could be improved. I can live with some slowdown in heavy scenes, but I’d like to push things better than they stand right now.

I need to measure things better, but there’s a quite a few steps involved:

converting values between 2’s complement fixed-point formats (4.4 -> 8.8 -> 16.8 with sign extension)
adding large numbers, using the carry flag to synthesize larger additions from smaller additions.
bounds-checking the entity position. This takes a fair amount of time, but is necessary since entities can move off the sides of the map.
converting pixel coordinates to tile coordinates
reading the actual data out of the tilemap
branching on the collision results
updating the position (if succcessful/unobstructed), or stopping the movement (if failed/obstructed)

Here’s some code for anybody interested.

The optimization steps to improve this could be messy, but hoping that my experience from writing programs for the Game Boy should help here. The Z80 is more powerful than the GB’s processor, so hopefully some of the more ugly tricks for performance aren’t needed, but I still have ideas for things to try.

The performance issues are also not even accounting for the added processing required to do entity-vs-entity hitbox collisions, going to need to do that soon too!

Lastly, I looked at adding a graphics loader for entities, so each room can have unique sprites depending on what is spawned there.

Half the sprite tileset is reserved for the player + common sprites like treasure chests, item icons, explosions, enemy spawn indicators (like Zelda 1’s cloud puffs), etc.

The other half is loaded between each room, allowing for entities to reserve their own sprite art, as well as the art for all their subobjects ahead of time. Then when the game is in action, no new graphics are required – simply spawn things that reference tile indexes based on what’s in the loader cache (cleared each room).

This approach of loading ahead does mean that any particular room’s full contents (including subspawns) must be known ahead of time, but it also frees up more performance to gameplay, since graphical loading when the screen is in use means sacrificing vblank time, and potentially causing things to split across multiple frames. This quickly becomes messy, and I’d rather avoid any of that for this.

There’s still a big list of stuff remaining! Not sure what I’ll do yet, but hoping I can continue to chip away at stuff on this.

These are all features from the original Wandering Magic game that still need implementing:

entity-vs-entity collisions
- support for obstructing other entities
- hit-scanning for bump attacks (think Ys, Xanadu, Fairune, and similar games)
- pushback on attack hits
entity movement/collision optimizations
hit flashes (temporarily colors the entity palette upon a hit)
HUD drawing
- health bar + exp bar
- health number + level number
- attack + defense + gem icon and values
- enemy health bar HUD (shows health bar and name + sprite when attacking an enemy)
- modal full-size textboxes (hides HUD and displays text until player confirms)
- short timed messages (uses same area as enemy health popup, disappears after time)
- pause menu - lets you look at your equipment, and use healing items.
room switching
- switching rooms when the player moves outside the screen.
- doors that switch rooms when walked over
entity implementations
- enemies (walk into to attack)
  - slime (weak, moves randomly, infrequently attacks unless the player hits them)
  - bat (alternates between flying around the room, resting, and flying at the player)
  - ghost (floats around the room slowly, occasionally moves toward the player)
  - dark mage (teleports around, shoots projectiles aimed at you)
  - snake (fast and aggressive, charges at the player a lot)
  - knight (slow and indestructible, you need a much stronger weapon)
  - skeleton (fast and runs around around, throws bones)
  - “red”/stronger versions of enemies (these have improved hp/atk/def/speed, and also grant more exp for defeating)
  - new enemy types
- destructible trees/doors
- chests (walk into to open)
- dialog npcs (walk into npc to see their textbox)
- player projectiles
- enemy projectiles
- shop items (stand over with sufficient gems, press action button to buy)
- mini doors (if the player stands in front of them with a ring, transform them to a miniature form, and allow the player to proceed through the door)
save system
- loading
- saving
- checkpoint tiles that trigger saving when the player presses the action button while standing on them.
game systems
- level ups and experience
- attack and defense stat gains on level up
- healing HP to full on level up
- equipment (automatic, each item belongs to exactly one slot, the next piece of equipment is always better than the last so it replaces it)
- magic (an infinite use item that spawns a projectile when the player presses the action button)
- healing items (consumable / limited use, restores player HP)
joystick input
support for more keyboard keys
title screen
game over

The above list started as an immediate task list, but quickly grew to a larger list of stuff to do while I was writing this. Heh, whoops!

Well, that’s all for this week! Hope to see you again soon!

Overkill

Another week passes.

I didn’t get as much progress done as last week, but I did do a few useful things.

I’ll start with the more interesting, which was to implement sprite flickering. This allows being able to see all the sprites on screen when there are too many on screen / too many per scanline, by alternating which ones are drawn each frame. This isn’t a built-in feature to the MSX sprite hardware, so games must manually cycle their rendering order to be able to flicker between the different sprites.

My flickering system works by adding a prime number to the draw order each frame, and using another prime to walk through the array. Basically: index = (index + PRIME) % COUNT; at each loop iteration to shuffle the drawing order in a nice way, so that every sprite can be seen.

Warning: Minor flashing Images (flickering sprite system):

Flickering sprites

So far I only implemented a sprites/scanline respecting flicker, but not something that will show everything if there was overdraw. But thankfully, I constructed the entity system to be 1:1 with sprites for now – it should be easy to do overdraw detection+cycling later if other things such as the HUD need it, or just reserve an entity for the HUD (sort of a hack but a cheap way to keep common sprite drawing structures).

I also optimized the movement code for Wandering Magic a bunch. Now it is possible to move 16 copies of the player entity at once without lag. Beyond that it’s still not perfect though (the earlier GIF uses 24 sprites and slows down when moving, 32 is even slower). Not sure what the next steps would be to optimize performance, if we want to do all 32 entities, but I think I’m okay with slowdown after reaching 50% capacity of possible stuff to put on the screen. and 16 entities is enough to have room for the player, a few enemies, and a few projectiles. A little slowdown is okay as long as the control still feels responsive and works in the player’s favor. But I might try to do a deeper optimization pass later.

Some optimizations I performed:

Reduce number of temp push/pop to the stack when not needed to preserve things.
Relieve register pressure by re-using more registers when their previous calculation is no longer needed. (This took a lot of careful reading of the code to figure out what registers were no longer live, something that liveness analysis in a compiler could probably do easily, heh.)
To multiply by 16, use repeated 16-bit self-addition (add hl, hl), instead of 8-bit shift left + rotate-left-with-carry. It takes less bytes and less cycles.
Comparison: Use inc/dec (which set the zero flag) to avoid overhead of comparing to 0 and 0xFF, when registers can be modified.
Sign extension: Use rotate/shift left by 1 (so the sign bit goes into the carry), then subtract a from itself with carry, to get 0 or 0xFF.
Use immediate load instructions when loading stuff into memory pointed at by index registers, rather than loading into a register first and then loading into memory

Some possible (much later) optimization pass:

page-align the tilemap array + tileset array
- possible because we load into RAM these can be easily located even if the source arrays aren’t aligned to the 256-byte boundaries
- page-alignment speeds up all pointer math, since single 8-bit adds / bitwise or can be used, rather than 16-bit addition.
replace 4.4 with 8.8 fixed-point. Speed is passed as an argument through static RAM variables, so direct 16-bit stores can be accomplished.
page-align entity array + remove use of index registers (probably not worth the extra complexity)
- indexed register usage is slower + takes more code space than indirected register pairs.
- register pairs don’t have random access, but page-aligning kind of gives you this back.
use shadow registers more effectively.

I also updated OpenMSX (and downloaded the debugger) and tweaked a few settings to have cleaner quality captures / better workflow. I fought with settings to get a nice clean integer scale render of the MSX screen. By default OpenMSX seems to like to use a bilinear filter over everything. Eventually after asking around, I figured out a some settings you need to open the F10 terminal in OpenMSX and set horizontal_stretch to 320, (which I couldn’t do from Catapult – OpenMSX’s GUI frontend) well as set scaler to simple, renderer to SDLGL-PP, disable all blur, glow, scanline noise, etc under video controls or from the OpenMSX F10 terminal. Anyway, way less friendly than most emulators in this regard, for getting a nice image without pixel blurring, when nearest neighbour is the standard in most emulators, but that problem aside it’s a useful tool. The earlier GIF was taken after fixing the scaling interpolation issues, making for less blurry screens than some of my earlier game captures.

The OpenMSX Debugger seems nice too. But I’m not sure if I might still fall back to the less-accurate BlueMSX because I’m more used to its debugger by now, and I don’t do anything that requires too much hardware accuracy (I hope). But I’ll definitely test with both emulators, and eventually true MSX hardware, I hope.

Other than that, I shopped around for a possible MSX to test development on. I found a Panasonic FS-A1, an MSX2 computer. It’s currently being recapped and then sent in the mail, if all goes to plan. There are probably cheaper ways to have gotten an MSX to test with, but not much to look for in North America, especially in Quebec.

I also ordered a MegaFlashROM SCC+ SD, which is a flash cart that allows playing software, and includes a reproduction of the SCC+ soundchip (that Konami used in their games, such as Metal Gear 1 - 2, Space Manbow, Nemesis 1 - 3, Snatcher, SD Snatcher, and so on). The MSX is definitely a more expensive system (though I think cheaper than the PC Engine), so hopefully after these two purchases I’m set.

I also bought a controller adapter to allow use of Atari/Genesis style controllers with the MSX, which is much, much cheaper than buying a nice Hudson Soft Joy Card or less comfy looking Panasonic 2-button controller. Controllers are weirdly expensive. So this adapter saves you about $100, heh. It’s not as essential since the MSX has a keyboard that can be used for games/programs too, but still nice this adapter makes it possible to get controllers.

Anyway, buying all of this equipment with the partial intent of enjoying games, but also to test my creations on the real hardware, since I can copy my MSX test programs onto an SD Card that gets interacts with the flash cart. We’ll see how that goes, and hopefully be able to set it up and take captures of the game running on the real console.

So yeah, that’s all for this week. Thanks and see you again soon!

Overkill

This week was slower!

I worked on Wandering Magic MSX some more. I’ve been porting the enemy logic from the original Lua code to Z80 Wiz code, So far, it’s going okay, but some things that are really easy arithmetic / comparison / math library calls in a high-level language, have to tediously be rewritten in assembly. Things like comparisons, while easy when comparing 8-bit values, quickly get messier when extended to 16 or 24 bits. Also, there’s a fair amount of counter variables that need to be split into multiple 8-bit pieces since (IX + dd) indexed addressing mode can only address a single byte at a time – so increments/decrements/adds/subs need to propagate intermediate results across the carry, zero, or negative status registers so that 16-bit math can be done. Eventually register pressure becomes important too, and book-keeping all of this can be tiring. I put it down for a moment, but I’m getting closer!

The “patrol” AI for the slimes, snakes and knights is a bit complicated, but the good news it will make a few enemies in the game work at the same time, once supported. Going to keep pushing this porting effort, and see about phasing features in a bit at a time, until each can be tested and verified separately.

Also, my MegaFlashROM SCC+ SD flash cart arrived in the mail:

Still awaiting my Panasonic FS-A1, but once I have that, I’ll be able to try things out on a real MSX system.

Recently, I was also a bit bored, and decided to write some data structure code in C. It had been a while since I wrote a hashtable/dictionary type, and so I made a hashmap implementations.

The hashmap does open-addressing / linear-probing and uses user-provided storage for tables with zero heap allocations during insertion. Instead the hashmap’s put operation simply fails when the table is too full. Like most hashmaps, it has best-case O(1) constant time, and worst-case O(n) linear time for lookup/insertion/removal, and it has O(capacity) iteration time.

I also provided some helper functions to adapt this to a dynamically-allocated + resizable hashmap, by adding an insert operation that will swap the storage to a bigger area on the heap as the capacity is reached. I also made it so hybrid static/stack allocation + dynamic allocation could be used in the same hashmap, by keeping the user-provided storage and only allocating when capacity of the current user-provided storage is exceeded. This allows optimization akin to the SmallVector type in LLVM, but for a hashtable.

In addition, I also did some library functions for a contiguous list data structure. Like the hashmap, they both serve as a simple wrapper over a plain array of data, or when wrapped with some dynamic-allocation, a resizable vector. And it’s possible to mix both pre-allocation + heap use again.

Stuff that doesn’t need the heap though, will be able to cleanly avoid this and build lightweight lists and hashes, or statically-allocated ones that are loaded when the program begins.

I also worked on a lightweight parser for a toy language. In some ways the scripting language could be thought of as similar to VergeC in some ways, where there are only a few built-in data types, no garbage collection. But unlike VC I was thinking to have automatic memory management through unique pointers to data + ownership/move semantics. Not going to allow refcounting, because the whole cyclic resource ownership problem it can cause.

Within the compiler I had the idea for some kind of language that could target a fairly “robust” set of features with a single-pass parse, but no Abstract Syntax Tree, instead convert directly to Immediate Representation, and either interpret that in a VM as a “scripting language”, or translate into something like equivalent C code, WASM or LLVM IR.

The idea being this language could be used to prototype stuff fast as a scripting language, and then build to C or IR when performance is desired.

I was also thinking of potentially digging out an old register-based VM (that I think I called “lilvm” heh, not to be confused with llvm) I had started, and see if any ideas from that mesh.

Anyway, I don’t really know what I’m doing yet, this idea is very much in the air, and so far I’m just building pieces that could potentially be used in a compiler. and coming up with ideas for a simplistic scripting language that uses strong typing, and compiles quickly.

My hope is something as easy to embed as Lua, but with no GC, C-like + simple to learn, and have the ability to turn into native code later. I don’t know if I’ll finish this, but having fun trying stuff out.

Also, outside that, I turned 33 and relaxed a bit. Happy Birthday me! Next weekend is easter so an extra day to relax again soon. Talk more next week!

Overkill

This week was once again pretty quiet, but I poked away a bit more at Wandering Magic MSX.

I also had the last-minute idea to finally throw together a Twitch stream showing some MSX development. It was fun! There was also surprisingly a lot more viewers than I expected, and I was pleasantly surprised by that! Thanks to anyone who watched, I sort of announced the whole thing over the easter weekend, so not a lot of advance notice. I figured the long weekend was the right time to give something like this a try, anyways. But still, about 7 or 8 people were viewing, which was neat and unheard of for most stuff I tried on Twitch.

It didn’t turn out to have any new code, and was instead a show-and-tell + hanging out and answering questions. It explains a little on the MSX, some of the pipeline I use to import graphics, some demonstration of the original PICO-8 game, some examples of some code I wrote for the MSX version. I had a bigger plan, but the script sort of got tossed out when there was more audience interaction than expected, so that made it sort of a more interactive Q&A + show-and-tell hahah, lesson learned for next time!

Anyway, if you want to check it out, and you don’t mind watching me ramble on for a while, here’s a 3 hour archive of the stream: https://www.twitch.tv/videos/976933676

(NOTE: some parts are loud due to some volume balancing issues near the beginning. So I recommend turning your volume down a bit lower around the Nemesis 2 gameplay. After this the stream gets better! There’s also a short dropout near the 2h30 mark or so I think, because my ISP. If I ever reupload this later, I might edit these out, but for now this version will have to do.)

Setting up for this stream took away a fair amount of time from actually doing new development this week. But I’m going to count this is as progress, since I did get to relearn how to setup a stream + use Twitch, and it might open the door to newer, better organized streams in the future!

Anyway, shorter update. Happy egg weekend!

Overkill

This week was once again very busy for me. I thought a little bit about what would be a good next step for the Wandering Magic MSX port.

In particular, I was thinking about trying to implement a subset of the enemy movement logic for starters, rather than trying to do a large line-for-line port of the entire behaviour at once. This way, progress is divisible into smaller, measurable chunks that can be tested and verified more easily. Trying to convert the entire function, although small and simple in Lua, is still a large undertaking to port all at once to assembly.

This approach sort of stopped development dead in its tracks a bit, because trying to port things was done without an immediate goal in mind anymore. And when my time is so limited, any loss of clarity is translated into wasted time before warming up enough again and getting somewhere. A consequence of this is, I suddenly feel less inclined to work on something because I can estimate how long it will take to get started, and know that the overhead to pay just to get started again is too much.

So, I need to come up with strategies so that it takes less time to resume the project. This ways it’s easier to shelve changes in the middle of something and come back. And I’ll still know what I’m doing, and I’ll know I have enough to make an attempt at the next piece.

I was thinking of a better idea:

Start with just getting slimes that move in a random direction.
Then try having them change this movement direction when a collision is encountered.
Then add an expiring timer to the directional movement, so they re-evaluate their direction every couple of seconds.
Then worry about collision detection between the player and enemy.
Then add stuff to do with damage resolution, then experience+gold on death, then knockback forces. etc.

The gist of this is, more itemized, short tasks with clear end goals. Rather than “here’s a function let’s rewrite the line-for-line equivalent”, without consideration for context and inevitably getting details wrong. This is easier to maintain things within my sparse free time/motivation on nights and weekends. And it might be easier to pick a specific task to show on a development stream, if I decide to do another stream on Twitch.

Nothing is perfect, and some tasks do require deliberation and planning, but by focusing on low-hanging fruit, and incremental tasks toward a bigger picture goal, more work can hopefully be accomplished that will push things ahead. And the goals/tasks can evolve in response to the changes made, rather than speculating and planning in a vacuum.

Anyway, so yeah would be worth giving this approach a try. Even just getting “pick a random direction and go in it” isn’t super simple, since it requires some code being already written, namely both movement code and a pseudo-random number generator.

In this case, I already coded a Galois Linear Feedback Shift Register for doing randomness, in 6502 and GB assembly, so the translation to regular Z80 should be pretty straightforward. These randomizers are pretty short in terms of code, and have maximal periodicity for its bit depth, so they’re kind of a nice balance between “good randomness properties” and “low implementation cost”. Better randomizers exist, but these are some of the easier to write.

To seed the randomizer, I plan to simply use a fixed seed on startup. But I will step the random sequence every frame, so over time the sequence diverges, depending on how many random values are sampled between gameplay choices. This means speedrunners can still get their frame perfect randomness (not that I anticipate any for my games, but…), and meanwhile normal players will still see enough random to feel that the game has some unpredictability/chance to its mechanics.

On top of just pure random, I typically like when games do stuff internally that behaves like shuffling a deck of cards and dealing those out, only reshuffling when the list is exhausted. Also, counting duplicates and choosing another action if one is used too much. this will also mask any crumminess in the RNG since it’s much easier to shape the random to something more pleasing to players. It prevents hitting the degenerate cases as much, by encoding rules that ensure each event has a finite number of occurrences, or a decided-once order that is then consumed.

Anyway, I got a little detoured here! adding randomness, and then adding a slime moving around that turns when a wall is hit, will already feel like a great start. Maybe by next week I can get this delivered. We’ll see!

Even while being blocked on my project, the MSX was still on my mind however.

I started playing around with taking some old art of mine, from a project called FROG EGG. I started imagining what it could look like in an MSX1 art style.

Then I made a mockup with a water background. My goal was to see if I could make a scene that had 1-bit sprites over a somewhat colorful background (instead of black bg), and be mostly readable.

Here’s the original 1-bit game this was based from:

frog egg 1-bit gif

The original game was a game for the Arduboy, a small Arduino-based AVR handheld system with 2K of RAM, that has a 128x64 resolution and a dpad + 2 buttons.

I ran into a snag though, in that, there is limited RAM + ROM space. Worse, if you go beyond a certain limit, the flash loader on the Arduboy gets bricked, and requires you to do the tedious work of factory resetting the device with a paperclip (easier said than done, from experience, even with the tools, because the reset button is inset pretty far and you need to hold it for a few seconds). Anyway, maybe one day I could make a small adventure out of the engine I did… it just got very precarious to make code changes which is not something I really wanted in a project in the prototype phase. (I eventually made an SDL wrapper of my engine for easier testing + sharing with non Arduboy users. That made things considerably easier, but still shelved things until I had a better plan formulated… And by then other projects came up on my radar lol… maybe one day)

Anyway, at the moment, I have no plans to make a platformer with this art yet. However, I thought it could be fun to create more of these little mockup scenes like the above MSX one, with some background pieces + some different sprites. Stuff like forests, caves, cityscapes, ruins, etc. Possibly more in the MSX style as well. If enough areas were done, it could be neat to explore making a game out of it. But until then, happy to just explore making some more art and keeping my pixel art from getting too rusty.

In addition to the above, I heard that Ludum Dare 48 is coming up soon! (the weekend of Friday, April 23 - Monday, April 26)

I might enter the Jam with some friends, since it allows collaboration, and they allow bringing in existing code+art+assets. I’m always busy when LD comes around, but in this case, I planned ahead and tried to clear plans to ensure a greater chance of successfully working on something. Never tried the Ludum Dare team jam, so it will be fun! Last compo/jam I entered was a few years ago, and the last collab was even further back than that. I’m really bad at entering these, but giving this a shot, since 1) the timespan is short 2) I have the time available 3) I planned time a couple weeks in advance.

Anyway, catch you later, hope everyone has a nice week! Next Gruedorf post will likely adhere to the later schedule I usually post in (Tuesday @ ~12:30 AM EST). This one was posted a bit sooner since I am busy tonight.

Overkill

This week was super busy again!

I overbooked myself to an extent, and had lots of things going on outside of work. It was nice to be moderately more sociable again, but came at the sacrifice of free time for projects work, and a need to recharge afterwards. I guess the weather was nice, so I wanted to enjoy this a little bit. (but done with respect to safety, social distancing, etc.)

Curfew also was rolled back to 8pm here, and that’s a source of stress and annoyance too, sometimes requiring me to rework my schedule to fit things in the hours we’re allowed to go outside. But yeah, still not a pretty situation here and case numbers continue to climb, and vaccination is still catching up to more groups. Waiting and watching the government updates, and staying close to home a lot, as usual heh.

I also got to work on my taxes… I had started a while ago, but didn’t make it too far, and with the Canada tax deadline approaching, the urgency and stress to finish that and make sure it’s done correctly is a weight on my mind lately. …But I didn’t finish that either! Some stuff about federal work-from-home benefits that I got stuck on, and I had to wait to ask questions to some friends/coworkers on that and a couple other things… But then I didn’t have a free night yet to resume where I left off. Taxes aren’t fun/passion project by any means, but they’re “work on something” that I needed to do outside of the hours of my dayjob! So it counts for “weekly progress”, maybe?

Speaking of dayjob, currently I work as a programmer for Tribute Games, and we’ve been working on Teenage Mutant Ninja Turtles: Shredder’s Revenge, a 2D beat-em-up with pixel-art sprites, and a spiritual successor to TMNT games like those for the Arcade/NES/SNES/Genesis. The game was recently in the April 14th Nintendo Direct, so that was exciting to see! Featured at 11:07 in this Nintendo Direct video –

There’s this gameplay trailer if you’d prefer something more compact format:

Some of the things I did programming on are captured in these trailers! However, as the game isn’t released, I probably am not allowed to say anything much beyond that. But still, it’s been very cool to work on. I’m happy and fortunate to be able to collaborate with great people on projects like this for my full-time job, and here’s hoping development continues to go smoothly for us. I still plan to keep my posts to primarily about personal things, but made a rare exception for this! (Also if you’ve never tried, Panzer Paladin are Flinthook are two other games that I worked on with Tribute, if you want to check them out! Plugging over.)

Aside from that stuff, I also had the idea to study some visual effects + animation captures, especially from SNES and GBA games, for research and analysis. I was thinking it could be cool to record some GIFs that first show a cool animated sequence or effect, and then do an analysis that decomposes the effect into its different elements + timings. If done well, this learning could even come in handy for creating new VFX creations for myself, and I could also share the notes made with others.

I was hoping to capture footage, in particular, from Super Robot Wars: Original Generations. This game series by Banpresto has excellent combat animations, and cool visual effects that are comprised of many parts. The Super Robot Wars games are are series of cool mecha strategy games that fuse elements of many popular mech animes, some original characters and even some cameos of Gundam/Neon Genesis Evangelion/etc in certain games.

The series has been around a long time, and characters from the older titles occasionally appear in newer games too. It continues to get new titles to this day, such as Super Robot Wars T (import-only but does have English version through EastAsiaSoft). They don’t seem to be released outside Japan and Asian markets very much anymore, but the few games we got in North America are a lot of fun to check out. We got the Original Generation series, and there’s also some weird DS spinoff RPGs like Endless Frontier: Super Robot Wars OG Saga too, not sure what outside of that we had.

One game we never got here, for the Super Famicom, was

which looks super cool, and has a fan translation by AGTP. I want to play this someday, but I’m getting sidetracked somewhat heh.

Anyway, here’s some SRW:OG GIFs of Kyosuke piloting his slow-but-powerful red mech Alteisen, performing Heavy Claymore + Revolver Stake attacks:

Heavy Claymore
Revolver Stake

I was hoping to dissect these into timelines, figuring out how much time they spend on each “state”, and also figure out what elements comprise these captured segments. I don’t have an animation or film background, so it would be explained more from the context of a game programmer who enjoys coding VFX. I would trying to think of things in terms of “systems” they use, their compositions, a possible flow between different distinct states of animation. I could see how different sprite particle effects, screen shakes, sprite shakes, palette changes, blinks, hit flashes, transitions, scrolling backgrounds, character movements, etc are done, and what they contribute to the overall effect.

I also wanted to capture a few attacks from FF6 for ideas from there too. I was hoping I could also compare it against earlier battle FF systems and show the smaller, less-obvious ways in which the battle animations were improved between game revisions.

Anyway, still gathering captures + playing enough of each game to be in a spot that can be captured nicely. Then next comes actually taking notes. Then making still diagrams / text explanations / or smaller explanatory GIFs. I don’t have time to do any of this yet, but maybe one day, if I keep pushing it along.

Well, this still isn’t what I hoped to have done this week, but this is how it goes sometimes. While I didn’t have the energy to do my game projects, I at least did other stuff that. And I wrote a lot of words here again! Maybe someday the stars will realign and I can continue where I left with things on Wandering Magic MSX, it’s still regularly on my mind even if it’s not moving at present.

And besides that, getting excited for Ludum Dare jam this upcoming weekend! Hoping to participate with friends. I was recently sent a doc by my teammate, that contained a tentative schedule + a guide to setting up with the tools that they’re planning to use.

Looking forward to be able to talk about how it went next update! Crossing fingers!

Overkill

Heyyy everybody, it’s egg here. Another week has flown by, and as usual things were very busy at work.

But not-so-usual, on this weekend, I ended up participating with some friends in the Ludum Dare 48 game jam.

We made a short game named “Frog”, which you can download here!

You’re a frog and you shoot bubbles! Descend the depths, find the treasure!

Credits

2bitcrook - art
cacciatc - programming
Kulor - programming
dustmop - programming
eggboycolor - programming

Plays in an NES emulator, or on a Famicom/NES/Famiclone that supports VRC6 sound expansion.

Try it out!

The game is pretty short, but considering the timespan and that it’s a real NES game, I’m pretty happy with what we manged to accomplish.

Thankfully, we picked the category of Ludum Dare that allowed not only team collaboration, but using existing code. Chris Cacciatore provided his NES-based game engine Veil to work with, and it was pretty fun learning how to write code with this toolchain. Chris, Dustin, and I collaborated on the coding side. 2bitcrook did the excellent pixel art for the backgrounds and frog + bubble sprites. The game supports VRC6 expansion chip audio, and kulor provided really cool music and sounds for that take advantage of this.

It had been a while since doing any sort of game jam/compo, so it was great being able to team up with a few friends to make this. It was a bit tiring at time, as I forgot how stressful fitting things into single weekend can be, but once I found the right rhythm and got used to working in the tools, things started to snap in place. It was helpful to synchronize on tasks each night + morning during the jam, as it let us brainstorm and change plans/limit scope when we were getting a bit stuck.

We originally had a few ambitious plans for the frog and the level layouts originally, with scrolling levels, multiple enemy types, and a tongue mechanic, but we quickly scaled it back. The final game has a bubble mechanic + static obstacles to navigate in order to reach the goal in each room. I’m happy the readjustment happened early enough that we could salvage the idea, and we managed to make some neat levels around it.

Hopefully people get a chance to play and try it out! It’s a bit buggy, as with any jam game (especially a crash at the ending, some sprite and physics glitches), but for the time allotted I’m pretty happy with the little platformer creation we accomplished.

Getting to jam was a nice way to recharge motivation for personal game projects. But on the other hand, it was also very exhausting to spend an entire weekend basically only working on that, with small breaks for food + walks. So that’s definitely not a regular thing I want to do. However, I’d potentially do another jam if circumstances were to work out again.

That’s all for this week! Hopefully soon I can finally dust off my projects and talk about something new.

Overkill

So, it turns out that my recovery time for doing a gamejam over a weekend is a bit longer than I thought! That sort of threw me for a loop, and I was running especially low-energy by the latter half of the week. Add to that, there was some sort of less-glamorous-but-necessary tasks at my dayjob and I was pretty exhausted.

I realized I’m still not back up to full energy to write more assembly code. But I worked on something new… that’s potentially for an old project! I’ve started working towards implementing a battle system mockup, that I could use for “nrpg”, my NES RPG described earlier on in my posts here.

I had originally wanted to maybe do a turn-based RPG that’s fast to navigate and has quick rounds. But about two weeks ago, I had a different idea for a completely different kind of battle system, with realtime elements.

I was thinking about a possible design for a “shmup RPG”, with the following:

real-time action resolution, like an action game (shmup, sidescroller, etc).
simultaneously command multiple units with an always-visible menu bar at the bottom that has 4 secitions:
- 1. party actions
  - formation
  - (??) autobattle
  - (??) special team actions
  - run away
- 1. 1. 1. actions for party member A, B, C
  - basic attack
  - skills
  - items - shared inventory. consumables, infinite use relics.
(??) actions can be interrupted up to their execution, and possibly during the execution of held actions.
each unit has a high-level state related to command progress:
- awaiting command
- moving into position (either a position relative to target, or an absolute position)
- charging (if applicable)
- executing
- interrupting
- recovery (if applicable)
each unit also has the low-level physical entity with its own state management. These states are things like:
- idle
- moving to target
- basic attacks
- different phases of magic / special moves being performed
- running
- hurt reactions
- death
entities move around in real-time according to these commands.
positioning matters (although movement is automatically controlled by simple AI)
simple pathing, no collisions/walls in the middle of battle arenas, so no need for A* or something – simply move in a direct path to the target.
attacks have designated ranges + targetting logic
animations could have attack/hitbox and vulnerability/hurtboxes that affect the collision
projectiles are spawned and move in real-time
evasion stats that could apply upon either incoming attack proximity, or as a last-minute reaction roll to a projectile collision.
timed reactions that can be selected from the menu as an enemy
would probably need a streamlined menu interface to really make this work, potentially limited slotted skills, like SaGa, Pokemon or some SMT games.
(??) waves of enemies continue to pass by - defeating them could yield bonus pickups that can be immediately gathered and used by party members. Like a shmup, they pass by and you can avoid their attack rather than fighting.
(??) barriers and protective fields that can placed on the map, to resist or negate certain attacks.

I decided to shelve the actual shmup aspect with ships and whatnot, but trying to see if I can implement the action elements onto a fantasy RPG, in particular my NES RPG. Reducing the scope of the above idea to make it easier to get started, and going to gradually see what is necessary.

I’m currently implementing a system to manage combatants, and an entity system that spawns/updates various things within the battle. Entities will be the building block for several kinds of things the move around and have sprites. They could have extra metadata associated with them depending on if they’re combatants, projectiles, traps, barriers, particles, damage numbers, etc. For now, I’m using placeholder art from some older projects, and trying to make stubs/placeholders for turn order, actions, etc. Going from there to block things in.

I want to see if I can mockup an action battle system using this. Failing that, I can reduce the scope to a turn based system later, and some of the same elements may still be useful. In any case, I want flexibility to handle different actions that could comprise of multiple states, and move and spawn sprites for VFXs. So even if the battle wasn’t real-time, the turn actions would have many elements that need to be spawned, updated and animated.

I’m using my simplistic C engine, vg, so that managing things in terms of 2D background layers + sprites is pretty straightfoward, and all the input handling and windowing stuff is already taken care of. Everything is written to avoid runtime allocations + loading, and instead uses statically-allocated global arrays, so that it will be easier to port the various subsystems to asm later. I could have used something like Love2D or Godot, but I just reached for a tool I knew would take not too long to get setup. And plus this engine follows some similar limitations I’d need to follow if writing a game on an 8-bit console, so it gets me planning for this ahead.

Going to keep working away at it if the motivation sticks!

Also something I hadn’t mentioned on here: I’m non-binary and my pronouns are “they/them”. For a long time, I just didn’t list anything so people could decide as they want, and I would only fill in people who were closer to me. But I realized I don’t really associate anymore with my assigned-at-birth + assumed-online “he/him”, and I wanted to do something small to update that.

I decided to list and update my pronouns on my Twitter a couple weeks ago. Before, my pronouns were never listed in my bio. For at least a couple recent years, I have considered myself non-binary but not openly. Without explicit pronouns, and because of my display name “Egg Boy Color” and my IRL name, gender assumptions were formed. And since I wasn’t yet open about this, I’d usually go along with the pronouns people used.

As for my choice of display name, Egg Boy Color was meant as a lazy augmentation of Game Boy Color but with “egg” instead of “game”. It was a console I was fond of and making projects for, combined with a breakfast item I liked eating. “egg” wasn’t used in the queer-coded slang sense, but I guess it could have applied in my case.

I didn’t think too much about the “boy” aspect, and still don’t. At companies like Nintendo and Sony, there was probably an antiquated marketing decision somewhere around the gendering used in Game Boy and Walkman, but the products themselves are inanimate objects with no gender, and went on to be enjoyed by people of all genders. So I dunno, I guess I figure it didn’t mean anything significant but is nonetheless part of the name. Even if my thoughts around my own gender might have changed since I chose my handle, I’m okay with leaving it as-is. At present, I still don’t mind my older nicks Overkill or Bananattack either. They’re legacy names, but simply retired them for sake of online searchability, and I liked eggboycolor better.

More recently, I mentioned my pronouns at my work, and in a couple other online communities. For me in particular, I don’t mind the occasional mistake, but I still would appreciate people’s cooperation in trying to remember my pronouns. In any case, I don’t currently have any plans to change my online handles or IRL name, or to change my appearance too much, but I can’t speak for the future. I have some opinions formulated and others still being figured out, and I’ll choose what/if I want to share. But probably not going into much more detail here, as I want this place to continue to focus on my projects!

I’m still new to being more open about gender identity stuff. Anyway, that’s a lot of words. It’s probably enough to know my new pronouns, and leave it at that! they/them.

Alright, that’s all for this week. Hope everyone has a good one, and see you again soon!

Overkill

Another week went by pretty fast.

I put more work into that battle engine prototype. It’s more or less executing the plan that was in-place last week, but not quite to the exciting parts of things yet. I completed most of the basic setup to do spawn a battle formation of player and enemy combatants. But there’s still nothing to really look at yet.

I also started structuring data things into different data structures. These are what’s in already:

Game Modes: different possible modes/scenes within the game, that may contain several substates and their own data they manage. Has an update callback as well as callbacks for entering and exiting the game mode (it’s also able to react on the mode that is being switched to/from, as needed).
Entities: things that can be spawned by the game, having a sprite and some data associated with them. Each has an init + update callback for the entity, along with some auxiliary userdata associated with the specific entity – this can be used to manage the extra data this particular entity type uses. Entities can be connected to many other subsystems like combatants, particles, projectiles, traps/zones, etc.

Potentially the same entity system can be used in both battle + other game modes, but with different entity types, but I’m only focusing on the battle for this.
Initial Stats: the constant stats data used to load the initial definition of a player or enemy.
Stats: a bunch of stats used to manage the live representation of something in battle. these have both current and max values of some attributes, such as HP and MP. Other stats are expected to not change. Like FF6, I was planning for players’ stats to be largely remain constant, and effects will be scaled by level, and augmented by external equipment / status bonuses. But there will be some things can affect the base stats of the enemy.
Player Definitions: the constant data defining the initial stats for a given character + other immutable properties for that character.
Player Instances: instances of a player that have been recruited/added to the story. Has persistent information that it carries along with them, such as the live representation of the player’s stats, their equipment, and so on. Some of these stats will possibly be copied into a secondary data structure for combat data.
Party: contains a fixed number of party member slots, each containing a reference to player instances that are in your current party. (there will probably need to be some sort of “reserve” party too, for keeping tabs of characters you’ve recruited and are available to the story at the moment. But maybe it could be as simple as a list of party members that are currently available – have that updated by in-game, and use that info for any party selection menu.)
Enemy Definitions: the constant data containing the initial stats for a given enemy.
Enemy Instances: the live representation of the enemy stats, as well as some mutable data associated with their current behaviour and tracking information (eg. possible targets, various counters and timers, managing more complex multiphase patterns).
Combatants: the pairings of a reference to a player or enemy instance, and reference to an entities. Eventually they will also contain some common state for managing the movement and command execution of a combatant. This mostly exists for places where the distinction between enemy and player doesn’t matter as much, such as targetting and common movement logic. Things like damage functions, decision process, additional stats bonuses (eg equipment on players, player-specific buffs, etc) can still delegate these decisions, but functions that take a combatant will wrap these where necessary,

Besides that, I drew a couple tiny placeholder sprites as stand-in characters. Now I just need a couple little enemy sprites and this would be pretty good!

I started replaying Star Ocean 2 since I had never finished that game (although I watched a friend clear it in university, and played through most of First Departure on PSP). I wanted to gather some ideas and research its battle system a bit, so it seemed like a good excuse to go back.

The combat so far is pretty enjoyable and fast-paced, with a fairly chaotic real-time combat. You take control over one party member and the others are controlled by an AI, which can be issued different strategies. Characters can attack with melee weapons, use special attacks (with varying speed/range/cost), shoot ranged magic projectiles that travel across the level, or cast special magic that freezes the gameplay during the effect. Enemies have different attack patterns and movements, and your units will run around chasing after the enemies and doing different things.

The game also has a lot of complicated skill systems and mechanics that let you craft different items categories (metalworking, cooking, etc), identify items, play instruments, paint, pickpocket NPCs, etc. Some skills combine to unlock even more skills, in addition to also having an effect on your character’s stats for investing in a particular skill. Some of the skills also eventually combine for even more powerful skills once you have their mastery, allowing you to craft some ultimate equipment or track down an optional boss or something.

So layered together, all of that it gets pretty convoluted, but it’s fun and eventually the game systems get pretty broken in your favor if you put enough points into things.

There’s apparently several endings and multiple characters you can recruit based on the story decisions you make, and other characters you decide to take/not take with you, optional Private Action events, etc. The final boss has a “limiter” that you can disable before the post-game if you want a much harder version of the final fight with the “true ending”. Neat.

So far I’m pretty early, just past the Cross Cave quest and on to the next story area, but I have a few characters in-party now. Also yes, the battle voice acting is awful and it’s great. Once I get further in, I’ll have a better impression of the “fully unlocked” battle system, as well as hopefully a better idea of how they change the combat to keep thing fresh as it progresses.

To make all this possible, I fixed my component-to-HDMI scaler setup (RetroTink 2X Multiformat) to work with PS1, PS2, and PSP stuff, so now I can play at my desktop computer. and potentially capture videos, and stream if I wanted. Not only for Star Ocean 2 but potentially other Playstation things. I also have Genesis component cables to test with my fixed scaler setup eventually. I had previously only the ability to do this with FPGA clone products of the NES and SNES, and a Rad2X cable for SNES. I guess the RT2X can also take composite if needed, so that gives an option for consoles that only have A/V but not component, even if the picture will be a bit fuzzy. Should be nice to dive into things more, in any case.

Anyway, I’ll leave things there for this week. Take care everybody!

Overkill

This week was quite uneventful, but I managed to get a little more progress on my battle engine.

I am currently at the point that I need to prototype some code that actually makes the entities move and perform attacks. I’m still figuring out where to go exactly… For now, I think a good way ahead would be to make everything driven by a really mindless “AI” controller. This could issue a basic attack move and picks a target any time they’re not currently attacking. Then try to prototype some logic for moving around the battle map, getting characters into range, and resolving the attack moves.

It would be nice if I can push past the “blank slate” problem things are currently at with this, and prototype a rough pass.

I also played more Star Ocean 2.

Getting to the mid-way point of the game and started mastering a few skills to allow breaking open more of the games systems in time. I now fully mastered Pickpocket-related skills on Claude so I can steal items from NPCs during Private Actions (if you do this with other party members in your group, it apparently lowers your reputation with them, so good to avoid that I guess! Pickpocketing your teammates within the town, when your group is only one member seems to be ok though).

Went back to a few towns and gathered some items – nothing too significant yet, but the occasional nice accessory + crafting resources. There’s one crafting item called a Rainbow Diamond I really want to steal off an NPC near the start of the game, but they make it extremely challenging to take. Pickpocketing an NPC only has one attempt each, so you pretty much have to follow a guide to know what each NPC will give you, and restart if you fail on any significant items. Also SO2 was made in the days before soft reset was common on RPGs, so every failure requires resetting the console, or getting a game over so you can reload your save. Eventually I decided I’d call it on attempting some of the rarer stuff for now in the interest of getting further in the game. The pickpocketing is entirely optional, but it yields some items that are hard to get for crafting ultimate gear later.

Eventually, I went back to the main story. I just completed the sword tournament in one of the cities. A story event forces you to lose in the final round to help show off how “cool” your rival Dias is as a sword fighter. I usually don’t care for these scenarios in RPGs, but it’s one of those ok cases where it’s one of those very clear 1-shot wipeouts where you know you can’t win.

I also went on a side-quest and recruited Bowman, a pharmacist who can rapidly throw handfuls of poisonous and explosive pills for special attacks. In order to recruit Bowman, you have to gather a rare herb from an annoying cave filled with lots of different herbs that you need to try collecting one-by-one until you find the right one. When I was doing the side-quest, my other characters gained tons of levels, because of the extremely high encounter rate and annoying branching maze-like nature of this dungeon. Using the “Scout” specialty set to avoid encounters at level 10 didn’t seem to have any impact. Some enemies in the cave provided a fun challenge but others were just annoying to hit and wasted lots of time targeting. But now that my party is all the way up at Level 45 for this point in the game, my magic characters can cast lots of heavy attack all spells to wipe most annoying encounters.

Bowman joined at Level 25, so it’ll take a little to build him up rest of my team. Either way, finally have a party of 4 members, so I guess once I recruit one or two other characters in the side-quests, my party will more or less set up for the rest of the game.

Starting to get a better impression of the game, and it’s got some definite rough edges in spots but still enjoying my time with it. I’m hoping now things will start to move faster in this as well, since this annoying dungeon side-quest is over + not needing to grind as much.

I also had a few ideas to work on my high-level assembly language Wiz again (which I use for making most of my personal homebrew projects). Before I can do that though, I really need to work on cleanup + refactoring of the codebase a bit. I have already made a few plans on how to gradually split up the compiler code into smaller pieces so that it can be easier to maintain, but it’s always a matter of motivation. Doing this sort of code-cleaning isn’t very rewarding to work on, but leaving it to work on new features makes those more tedious than they could be, and makes the mess grow more.

None of these are impossible changes, it’s really just sort of burnout and dread thinking of going back to this part of the code. Even though I like the process of creating a programming language and making compiler tooling, it’s a different story about re-organizing the internals. It takes a lot of building up motivation to do these sort of tasks, for me, sometimes months away from the project until the right things align. Anyway, it’s been on my mind more again, so it might mean I end up going back to it again. We’ll see.

Anyway, that’s all this week. See you again soon!

Overkill

I’m late for my usual weekly update. It was a long weekend (Victoria Day weekend in Canada), so this threw off the normal schedule a bit. The weather in Montreal also warmed up, which was pretty nice outside. Indoors the humidity made things a bit less pleasant, and I had to get out a fan.

I’ve been struggling to make much headway on anything programming-related outside of work in a little while. I’m not sure why, but I guess sometimes this happens. I was a bit tired with my current projects, so I decided to start on something a bit different.

I had the idea to write a music engine for my homebrew NES games. I wanted to support FamiTracker, since it’s the most popular music tracking software for composing music for the NES.

Before going down this road, it’s useful to evaluate what’s already there. A few other music popular NES engines exist that support importing from FamiTracker, such as FamiTone (or the older, original version here), GGSound and Pently. I’ve used Famitone and GGSound a bit, but they lack some features for making richer audio tracks in some regards. Never had a chance to try Pently.

There’s also another tool FamiStudio, which is a separate piano-roll style desktop music editor. It has FamiTracker import, and supports targeting directly for its own improved engine fork of FamiTone. But from the page’s own description, it sounds a bit closed to outside contribution and it’s a larger codebase, so I’m bit unsure about it for this reason.

Anyway, I looked at what was there but wanted to try writing my own thing. Hopefully something that has easy integration into my own projects, and has good support for some of the tracker effects that FT has available.

I also wanted the potential for supporting expansion chips, to approximate the sound capabilities in other game consoles, such as the MSX, PC Engine, or Game Boy. It wouldn’t sound 100% the same, but it would give a way to test these things out at least.

(MSX has on-hardware stuff like Trilotracker, and Game Boy has on-hardware LSDJ, and Nanoloop. But these are more strictly for writing and listening to music when nothing else is happening. Not meant as much for gameplay when the CPU is needed for other tasks too. There’s also Deflemask or XPMCK, which support a bunch of platforms but also don’t really have drivers that work great for a game. GB also has Paragon5 Tracker, but it’s ancient and clunky. So in light of all this, something like using FamiTracker to approximate ends up sounding like a good possible alternative.)

I’ve previously worked on a a couple attempts on different music engines, but these always relied on notation similar to MML (Music Macro Language), or byte-code interpreters that used command opcodes for notes+effects. In these cases, I was writing the engine, but didn’t have easy integration to a usable music editor. They had sample data or text-based formats that they could convert, for the purposes of testing what sound engines were capable of, but nothing easy to actually make music with.

There was also Bleep, an unfinished piano-roll music maker I was working on that ran on the GB hardware directly. While this was neat, and I want to continue with that someday, it’s less practical for game music making, and was more meant as a fun way to make songs on the go. For my current idea, I wanted something with easier integration with a desktop music editor that does chiptune music. Different goals.

Alright. So this time, I wanted to make a project that took things in the other direction – rather than engine-first like usual, I decided to try doing an importer tool first, then a converter tool. This way, I could write a music player that can interpret the parts of FamiTracker song that I support so far. This would allow making music in FamiTracker from the beginning. I can incrementally add support for more things, while being able to compare the output from my engine vs the FamiTracker. I can also use my past sound engine work to fill in some of details later when I get to that point.

FamiTracker’s normal file format is a binary and seems a bit annoying to handle. Not impossible but probably would take some extra time to get right. Thankfully, it also has a text-based file format that it can export that is (comparably) much easier to handle. Sadly, according to its own documentation, this export is not guaranteed to stay stable between versions… However, it turns out FamiTracker doesn’t change all that often anymore since it lacks active maintenance for the official version (last release in 2015). I also find it unlikely that they would abruptly throw it away fully, since lots of external music engines + converter tools depend on the format working a certain way now. So yeah, text format it is!

The format documentation is included in the .chm help file that comes included with FamiTracker’s installation. I couldn’t seem to find a direct online version of the text format specification on their wiki or website.

However, through a roundabout way (a github page that can link to .html files in github repos), I’m able to link to doc for those interested: https://htmlpreview.github.io/?https://github.com/HertzDevil/famitracker-all/blob/master/hlp/text_export.htm

This format is a simple enough: text-based format with line-based commands consisting of space-delimited tokens. But there’s some slightly tricky business comes with how it handles strings, which need some escaping. Otherwise, not to much to say about the format itself. The commands of the format do stuff like define tracks, instruments, macros, etc, piece by piece, as each is read and interpreted. It’s kind of unfortunate that it’s a custom text format, rather than just using JSON or something, which would make it a bit easier to integrate without other tools needing to write a new parser. But what’s there is still pretty nice. Parts of it can be more easily be human-read, and the music patterns themselves essentially look very similar to how they’re written in the editor, so it’s easier to inspect.

I originally wanted to write this tool in C++, but kinda regretting the decision to not just phone it in with a Python script or something instead. Writing anything that handles I/O and text handling with nice graceful error-handling always takes some work to get right. All in all, it’s pretty much the equivalent of split() with maybe a couple regexes, but I ended up settling on a simple character-by-character scanner instead since I could have finer control over the memory usage and more easily do input validation and errors with line info.

Aside from the text input stuff, I made decent progress on some data structures representing the in-memory representation of a FamiTracker document. That part was kinda fun to work out, going backwards from the description of the various commands in the export spec. I’m sure if I need I can read the FT source later too, but I wanted to “clean room” things a bit for now.

Anyway, I did manage to write most of a parser + a bunch of struct definitions for the FamiTracker’s document structure, individual tracks, patterns/rows/columns, instrument definitions, macro sequence definitions, etc. Didn’t quite get to the end, but close.

If I can manage to get this parser done, I can start selectively outputting parts of FamiTracker songs from this tool. Or potentially toss it out, treating it as an exercise only, and maybe think about doing it as a Python script after all heh. We’ll see.

I ended up returning a little bit to development plans for Wiz, too.

I created a private Git branch of the source to allow me to try stuff. I like open development most of the time. However, for big changes, it can be annoying to make sure breakage doesn’t happen at every step of the way, once people are able to see. And I’d rather make intermediate commits that break things and fix it. I could make a separate public WIP branch, but for this, I don’t really feel there’s much value in people seeing my in-between meanderings and prototyping that I might toss away or never finish. So private repo it is.

The project doesn’t yet have unit tests, so that’s one thing I want to fix. I also want to work on splitting the compiler into smaller modular pieces, which I’ve probably talked in another post before. Decided to just rip the entire compiler code out for starters, and phase it in piece by piece. This way, I can put more thought into where things should go this time around. I can potentially I clean up some inelegant things, and can review and make fresh notes about what to improve along the way.

I’m still not sure about it, but I’ve long hoped to add features that can define platform backends directly in the .wiz language. Wiz already has it so that every platform provides definitions for register sets, addressing modes, instructions, and their encodings. Right now these are done directly in C++, but if these could be moved, that would make maintaining these bindings easier and allow new ones to get integrated too.

With a little bit more work on top of that, it could even possible to allow mixing raw assembly with Wiz-style high-level code. This would allow interop with code written in other assemblers, allowing the inclusion of libraries that weren’t written in Wiz, without needing to port their sources.

Another thing is that, Wiz currently targets binary opcode formats directly from its IR, and doesn’t have any text dump of the final program code. Having this feature would allow easier verification of the code generated for bug-checking the asm and potentially for writing some functional tests of the compiler.

There’s also some language warts I’ve wanted to iron out, this gives a chance to potentially revisit that in the process.

Lastly, I got a new laptop, an Asus Vivobook Flip 14.

It just arrived after the long weekend, and so far it seems pretty decent.

The 2-in-1 laptop-and-tablet aspect wasn’t needed but it’s a nice extra. So far, I managed to uninstall the bloatware, disable a bunch of Windows 10 features + apply registry edits, and install a bunch of basic applications for getting development stuff done on it. And there’s Chrome, Steam, Spotify, etc.

The half-size arrow keys are unfortunate, but unfortunately all Windows laptops I looked at had some drawback or another for my personal preference. Also, this laptop seemed to have some of the better specs for comparable things of its price. And didn’t do things like remove the headphone jack or only have one USB port.

Overall, pretty happy to have a functioning laptop again. It’s been a long time since I’ve been able to do development away from desk.

My previous laptop was from 2013. While still kicking somehow, it has a keyboard that’s barely longer usable. They don’t make it easy to clean the keys without breakage. It’s been like that for about 1.5 years now (basically keys died right after xmas break in 2019). So it’s made me do all development work exclusively at my desk. With the pandemic requiring my home desktop to become my full-time workspace, being able to do projects on a different computer, away from my desk is quite nice.

I’ll probably keep my old machine as media box until it completely runs into the ground, since it’s still possible to type short parts of a url into a browser and then autocomplete and click on things.

Okay, see you next week if I can remember this time! Now, to try to sleep again.

Overkill

Hey everyone. This week flew by fast. I did my last update two days late, so there’s been less time than usual between my posts. I almost missed my post tonight, even with things back to normal + the usual notifications I have going for posting on Monday nights.

I got a little spark of motivation to resume work on Wiz, my high-level assembly language. I’ve been working with the private repo branch I recently re-made, so I can take my time to mess around with these changes and not block the public upstream.

I decided against the ‘delete code and re-add’ approach when it came to refactoring the project. This is because it would mean losing the ability to run the program against example source code until the refactor was completely finished. These examples are currently the closest thing I have to tests for the compiler. And because they’re fully running programs, not only do I get to see if the compiler works, I get to look at the resulting program in an emulator to see the product of my work. So anyway, that’s why I don’t think that a clean slate fashion of rework would go very well, without putting a lot of old code back, which would inevitably probably end up rushing things back to the position they already were.

(Well, there are some functional test suites, but they aren’t as interesting to run right now, since they mostly test the codegen pass. I eventually will rerun these though. And I also want to write seams to allow for some proper unit tests…)

Instead of the “destroy-and-rebuild”, I decided that I would start by moving existing code into separate smaller subsystems: lifting things out of the compiler source and into their own file. For a good place to start, I moved all code related to scoping, name generation, and symbol resolution. Thankfully, all of the scope-related stuff has almost no dependency on the rest of the compiler, so it’s an easy candidate to move. This is just moving things over mostly so far, but I did make some minor improvements. It is giving me a chance to re-read the various pieces and see things that could be done better.

Things still have a long way to go, but if I can have like 4-5ish smaller systems, rather than 1 giant intertangled mess, I’ll feel better. For some reason, doing this sort of cleanup on a personal project seems like such a chore. I guess because it’s an unpaid project, I’m the only maintainer, and there’s limited appeal to work on a compiler in the first place. For me, I have to be in a specific mindset to want to do tools-related work. Also, I don’t want to break the publicly-available upstream version of the code, since a few people are using it. With the project already “released” kind of, it feels more difficult to continue do new things to it, even though I’m in charge of its development.

With a compiler especially, people tend to expect minimal breaking changes to the language. and any change needs to be considered carefully so it’s actually an improvement, not a step backwards. Anyway, this is why getting to this point has taken a few failed attempts at wanting to get around to fixing the project, months apart. Having a private dev branch has made things feel a little less risky at least. I also have less anxiety in general around making bad choices, since I can at least squish any bad decision before it becomes a part of the public release,

If I stick with this and finish on this, I would maybe like to take a crack at improving some of the data structures in the compiler, and improving some of the patterns for memory management. The data structure things would be mostly a learning exercise, but also to have better control over the performance characteristics and share the same data structure implementation code across platforms, rather than depending on STL vendors.

In the case of the memory management, using arenas would allow things to be packed closer in memory, making them more cache-friendly. It would also potentially also allow more stuff to use cheap non-owning pointers to each other, and potentially avoid cloning in some places that currently need to do this. Cloning was needed preserve the current model of unique pointers + immutability, but in some cases, non-owning sharing would be possible. Shared-ownership pointers were deemed too messy a model since I didn’t like the unclear ownership situations it could create, and I didn’t want to pay the cost of refcounting, nor risk the possibility of a circular references being formed. Arenas are nice because they’re simple to reason about, efficient to allocate, and can even speed up destruction if I refactor things to ensure everything is trivially destructable, since I could just release entire blocks of things at once.

I’d also like to see if I could make the code for describing the various instruction sets entirely data-driven – it’s mostly there but there’s a few things that could use some work. If this were done, potentially it would open the door up for Wiz to define platforms and their instructions directly within .wiz. This would let people use use Wiz to target any architecture that there are rulesets defined for, even custom virtual machines too, if desired. It would also lessen the time spent maintaining these platform bindings, since they wouldn’t need to be baked into the compiler anymore. I’d probably bundle some bindings built-in but allow superseding them with newer versions.

I thought adding compile-time arguments to inline functions would also be cool. It’s an idea I had since early on, but something sorely missing. Writing specialized code without going for the manually unrolled and edited copy-paste approach is appreciated. Since Wiz allows constant-folding of if statements, there’s also the possibility for compile-time specialization based on the arguments passed. Some people have even hacked in their own preprocessor pass to wiz just have #define macros heh. I had originally held off doing this feature so that I could evaluate how to do things, but I think I have an idea of what I’d like to do.

It could also be neat to structure the tooling such that it could interact with the Language Server Protocol used by modern code editors such as Visual Studio Code, Atom, Sublime, etc. If such a service was made, potentially tool-guided autocomplete and refactoring would be doable. My compiler needs a bunch more work to support that though.

One thing at a time though, let’s see if I can keep cleaning things up and maybe spark more motivation I’ve been missing for a while. See you next time!

Overkill

Howdy. Another week, another update!

Since last week, I continued some more refactoring on Wiz when I got some free moments. I kept working on splitting the compiler code into a few smaller parts. The division right now is a bit arbitrary, but it’s good to start somewhere.

A lot of functions that used to be instance methods were rewritten to be free functions instead. One nice advantage of this, is free functions can’t act on private state (without gross C++ features like the “friend” keyword). As a result, more fields become public, and data structures are a reworked to be bit more transparent.

Another nice property of using free functions, is that it’s possible to move these functions around between source files without problem. Also, for places where the information hiding is useful, or the implementation details don’t need sharing, some things that were previously private data + functions within Compiler could be put into a file-local anonymous namespace instead.

For now, some things were rewritten to take a common CompileContext, containing various smaller systems and state across the course of compilation. Eventually the big mess of data members can be organized better into smaller logical parts, and some things that take a full-blown CompileContext could take a (possibly const) reference to a specific subsystem they need. This would make dependencies and their usage clearer to tell from the function signature. Some of the reworked functions already do this, but still working out the details for others.

I’m currently working on just moving the code, with minimal changes inside the actual function body. But I intend to clean things up more when things are divided further.

As an example of something I want to clean up better, reduceExpression is a function that reduces an expression tree into a smaller form. Given a abstract syntax tree for an expression with no extra information, it does semantic analysis and produces a partially-folded expression tree with type-annotation on the nodes. It can also take a previously-reduced tree and attempt to reduce it further. For example, expressions with references to constant symbols that can’t be fully resolved during the initial compile, but can be figured out during the final code generation.

Anyway, turns out this function is like 1000+ lines of different cases + logic, basically it sorta outgrew itself and code just kept getting added there. This could easily be split up into a bunch of smaller functions. Splitting up the big mess has made it easier to spot the clutter within.

There’s also the issue of naming things better across this codebase. For now I just sort of called things the first name that came to mind, but there’s room to rename and make it clearer. Ideally naming should be short, concise, and clear, and consistent where possible.

Trying to not fall into the trap of nitpicking over minor things forever. Better ideas might come up when more important changes are made anyways. And there’s a point where it’s “good enough”. I just want to get it far enough that I hopefully won’t feel that I’ve coded myself into a corner anymore. It still will probably not be code I want to look at or work on every day. But having things more organized still goes a long way.

Anyway, more work to do. Hopefully I can keep the motivation to continue this. I’d really like to finish this cleanup so that it’s easier to add testing, do bugfixes, add new features, etc. like I’ve been talking about. If I keep working a little bit at it every couple nights or so, I can eventually get this into a more maintainable state, but it’s a matter of having the energy and attention for it.

In other news, the MSX2 computer I ordered way back in March finally arrived. The model I bought was the Panasonic FS-A1. The box art on this is… pretty incredible hahah. The orange-on-black look of the console system is very cool too.

The FS-A1 with glorious box art
The unboxed system

Here’s a brief video I posted on Twitter of it running!

Unfortunately, the phone didn’t pick up the audio from my TV (unless you crank up your volume), but the flash cart also supports the SCC expansion audio chip that Konami games for the MSX used. Nemesis 2 has a solid soundtrack, and it sounds excellent coming out of a real console hooked up to an old CRT.

It doesn’t have all the bells and whistles of some of the last generation MSX2 systems, like the MSX2+ and Turbo R, but I can live with it. This console was in great condition and in my price range. Meanwhile the later revisions are exceedingly more rare and expensive for only a handful more titles, and only slightly better specs. I think the middle-of-the road option is good enough since it supports all the MSX1 games, and most MSX2 games work ok except for a few exclusive releases for the later models.

The seller also had the option to add some kind of RGB mod, but I don’t have equipment to handle that at the moment. And from reading online it sounds like I maybe dodged a bullet there. They didn’t appear to have great feedback online for this particular thing.

Meanwhile, they took a picture of the recap they did before sending, and thankfully the work they performed looked okay, from observations as a layperson. No damaged traces or anything like that which I could notice, so hopefully all good on that front. Always a bit risky to get unknown eBay sellers to do work on a system, but thankful that it all looked good. Especially since I found out this information by searching the seller name after the fact.

In any case, it works great when plugged into my old C64 monitor over composite. I could also probably plug it into my RetroTink 2x and use that for capture. Even if the signal isn’t going to be super crisp, it’ll probably look good enough. I could possibly find a reputable modder to install something down the road if it’s possible to have a cleaner video output, but not in any hurry for this.

There’s more encouragement to get back to MSX development, now that I have the ability to run and test stuff on the real thing. I can continue my port of Wandering Magic and see the results on an actual console, instead of just in an emulated environment. Or maybe tinker around and work on something else that seems fun to make, too.

That’s all for this week. Catch you later!

Overkill

Quiet week.

Work was busy, and the particular feature work I’m doing right now is really not that enjoyable to look into, but necessary for the project. Trying to motivate myself to finish this, so I can look at something else, but there’s a ways to go with it.

I did a little more coding on Wiz, moving things around and attempting to clean things up. Still lots of mess to still go with this, but starting to see the other side of it. The intermediate work is kind of unenjoyable to do, but trying to keep working through this. Lots of stuff that probably needs to get organized better, and it’s painful to look at. Trying to split things out into the smaller subsystems, and fix up all the dependencies between the different subsystems, so the whole thing can compile again. Even if it means a mess in the meantime, hoping I can close this out at some point.

Again, it would likely be way easier to do this cleanup work if it was tired to a direct feature or something, but unfortunately, I put off cleaning up the technical debt for too long. Making it hard to add features without growing the mess, and hard to adapt much of what was there without a refactor in the first place. But this is stuff I’ve said already at this point.

I’m kind of discouraged how slow this is going, but considering I maybe only spent a few hours total, spread out across days, it’s understandable why this is a slow and tedious process. Still, if I can finish up this initial pass at splitting everything up in some form, then I can worry untangling things. It might be easier at that point, because it will be possible to divide-and-conquer the pieces to tidy up, without impacting the others.

Also, on Sunday I got sick. Originally I just thought my apartment was stuffy, or maybe I was dehydrated or something, because it was really warm out. But I drank plenty more liquids, and the symptoms persisted overnight and during the rain. I don’t have a thermometer, but it’s 17 C at night and I didn’t really feel this at all. During the daytime today, I actually thought I was better, but tonight the fever-like aspects returned. I’ve been isolating at home and have the windows open and fan on. I currently only have 1 of 2 vaccination shots, so there’s always a potential risk.

I had originally planned to have a stay-at-home vacation at the end of this week, but I’ll most likely need to cancel, so I don’t burn my vacation days as sick days. Unless I get very lucky and this thing resolves itself by the morning or something. I have a feeling I’ll need to get a test and self-isolate at home more, which is not fun.

Wishing that this blows over quickly and crossing fingers that it isn’t what I think it is. Hope to talk more next week.

Overkill

My week was pretty rough due to falling sick. My fever had worsened, and my condition progressed to the point that I went to get a COVID-19 test. The tests came back negative, thankfully, but it still took me a few days to get better. I still worked from home in the meantime, but otherwise rested and drank lots of water, and didn’t do much.

By the end of the week, I had finally recovered. I managed to squeeze in a little bit of time towards working on Wiz. I continued pushing along the refactor, finally lifted out all the compiler methods into free functions that act on a CompileContext (or its sub-members) instead. I am still not 100% on how to organize things but at least now all the data managed during compilation is contained in a simple struct with public members. This makes things easy to access by various passes/compilation subsystems. and functions that act on this data can now be moved around/regrouped a lot easier. Eventually, the actual data for things like scope registry/attributes stacks/etc could be refined to use custom data structures with a better-organized set of operations. But at least this simple move-out approach is able to work with the existing code with only a few edits / adjustments.

I’m happy to be making some headway on things. I just need to fix a few more pieces, and hopefully the rework will compile again. I am hoping if I can get past this part, and tidy things up a little bit, I can bump the code on Github with the changes. Also, starting to feel like I might also have the capacity to work on game sideprojects again, after feeling drained and lacking motivation for a while.

Anyway, as rough as the past week was with being sick, overall I’m doing much better now. And two short weeks in a row ahead: one for Quebec’s provincial holiday, one for Canada Day. Maybe this will give a chance to recoup a little bit more, and dive into more sideproject work. Let’s see how it goes!