https://solhsa.com/oldernews2025.html#ON-FILE-FORMATS

Sol_HSA
# Search # Email # Mastodon # Discord # Ko-Fi # Github # Linkedin #
 RSS

 General

  * News/Blog
  * Who?
  * Stories
  * Tutorials
  * Breakdown
  * Links
  * Stuff

 Speccy

  * 48k spectrum
  * ZX Spectrum Next

 Downloads

  * Code
  * Demos
  * Games
  * Audio
  * Misc files

 Minisites

  * SoLoud
  * FoodSofia
  * GalaXQL
  * Soundex broken telephone
  * Thesaurus broken telephone
  * Advent of Code GIFs

 Meta

  * # Search
  * # Email
  * # Mastodon
  * # Discord
  * # Ko-Fi
  * # Github
  * # Linkedin
  * # RSS

Blog

On File Formats

May 19th, 2025 (permalink)

I've been meaning to write something about file format design for a
while, and might as well do a blog post. This is pretty unorganized,
in a moment thing, so take it as it is.

So You Need a File Format

There's a few questions you have to ask. Different files have
different goals, after all.

1. Does a file format exist for this yet?

It's a good idea to check what the prior art is, as it's more than
likely that there's already something out there that fits the bill.
Let's say you need an image format. There's tons of image formats out
there, so one of those might already solve your problem, possibly
with tool integration and plenty of library code to boot.

But it's also possible you're in need of a image format for something
really specific, like a 4-bit cpu driving a custom LCD with a strange
pixel order pattern. Hitting that with jpeg is probably not the way
to go. So you need something custom. Assuming, again, that the
community for that specific use case doesn't already have a standard
format. That you don't have a problem with.

There are cases where you may want to use existing complex systems,
such as SQLite, or something that automatically serializes your data;
some programming languages even provide functionality for that. There
are pros and cons for these approaches, so you may end up having to
design something from scratch.

2. Does it need to be human readable?

Parsing text is a pain, so again you need to ask whether you can get
away with json, xml, ini or some other commonly used format, and just
design fields in it. Just picking one of these formats isn't the end,
though, since the structure of the data matters.

There may be a reason you don't want to use those formats, in which
case you need to make decisions based on your specific use case, and
the reasons why you can't use a common format. In this case I would
concentrate on minimizing potential errors, both technical and human;
Make the format as easy and foolproof to parse as possible to avoid
plenty of headaches ahead (and you may want to avoid fscanf for its
quirks), and make the human-editable data as unambiguous as possible:
if you need true/false values, pick one set and stick to them. I
would use 0 and 1 as it's harder to typo "0" than "Disabled".

3. Chunk your binaries.

If the data doesn't need to be human readable, it's often way easier
to make a binary format. A common structure for these is a "chunked"
format used by various file formats including 3d studio .3ds files,
autodesk animator .fli/.flc, EA's .iff (famous on amiga), Apple's
.aiff, Microsoft's riff (including .avi) and countless others. The
basic idea is to define data in chunks, where each chunk starts with
two standard fields: tag and chunk length. Chunks may also contain
other chunks, meaning your whole file can be one chunk that then
contains other chunks inside it. You can find the whole file size by
just checking the root chunk size.

Some people call the initial bytes "magic numbers", used to identify
the format. I just call it the tag of the root chunk. Having the
whole file be a chunk has the additional benefit of being able to
concatenate your files into one bigger file, if need be.

As an example, let's say you have a file format about cats. You might
start your file with a "cats" chunk, meaning the first 4 bytes in
your file are "cats", followed by 4 bytes describing the whole file
length. After that you might have some header fields such as format
version, how many cats are described in this file, maybe a field for
identifying what software generated the file, etc. After these, you'd
have a bunch of other chunks like "toys" for different cat toys
(again starting with the tag, followed by that chunk size), "name"
for ideas for cat names, "colr" for different cat color schemes, and
so on.

Using a chunked format allows chunks to be in different order, and
there may be optional chunks, possibly added in later versions,
adding backwards and forwards compatibility. Which leads us to:

4. Allow partial parsing.

Whether your data is in text or binary format, design it so that it's
possible to write different tools that read your format, that may be
interested in different parts of the file. If I wanted a corpus of
8-bit palettes used in .fli animations, I could just write a tool
that skips all chunks until it finds the palette chunks and extract
just those.

5. Version your formats.

It doesn't matter whether you never, ever, ever plan to change the
format, having a version field in your header doesn't cost much but
can save you endless headache down the road. The field can be just a
zero integer that your parser ignores for now.

6. Document your format.

Even if it's for personal use, write a specification of your format,
in a way that someone else than yourself could reimplement tools for
the format. That someone else may be yourself in 20 years, and I
guarantee you, you won't remember all the details. Be as unambiguous
as possible.

7. Don't include fields just in case.

Most formats out there have these. Save them for a future version of
the format. You may be tempted to pad out the file header with
reserved field for future expansion, and that may be valid. However,
it's cleaner to have a field in your header that states where the
first sub-chunk starts; that way you can expand your header as much
as you like in future versions, with old code being able to ignore
those fields and jump to the good stuff.

8. Consider the target hardware.

If you're targeting a small device (something embedded or maybe
retro), the target hardware may set a lot of restrictions on your
file format design. Anything can compute anything, given enough time,
but if your device is based on big-endian 16 bit integers, it's
probably good idea to stick to that instead of making all fields 64
bit little-endian. Other limitations may also apply, such as memory
sizes and alignment.

9. Compression.

Depending on your use case, compression may be a factor. Should you
compress your whole file, or do a chunk at a time? Are you planning
on modifying the data later on, or is a created file always
considered read-only? How much memory does it take to store the whole
decompressed data? Do you need to seek within the file, or is it
always read from beginning to end? How fast does the decompression
need to be?

If speed is a factor, it's possible that compression actually speeds
up reading, as there are compression algorithms out there where the
time saved reading less data from storage is greater than any
slowdown caused by decompression.

10. On filename extensions.

You may want to look up whether the filename extension you're
deciding on is in use already. Most extensions have three characters,
which means the search space is pretty crowded. You may want to
consider using four letters.

Deep Fishing

April 17th, 2025 (permalink)

It's been a while since I last wrote anything for the zx spectrum, or
done any creative programming in general. I've concentrated more on
my steam backlog and reading books and stuff. I had played through
Dredge (as much as it's sane to do so), saw that Ludum Dare had
"depths" as a theme, and, well, the Comp.Sys.Sinclair Crap Games
Competition rolled around, so I figured.. I might do a fishing game.

I started off with a design document. Here it is, verbatim:

space to fish - line goes down, release, line goes up: get depth value (0..32)

16 or 32 different fishes:
- 4x4 random pixel pattern
- 1-8 rando length
- random depth offset
- valid depths random(1,2,3,4, 6,8,14,20)

Each depth has chance to get fish that is PoT available fish + nothing
(if 2 fish available, always get a fish; if 5 fish available, 5/8 chance to get a fish)

Clients come with shopping lists of 1-8 fishes (with pictures)

Player discovers fishes and has data of what fish they have caught at what depths

Game goes on forever

I did not follow the plan quite one to one, but the gist is there.
Next up, I did a mockup in my old Photoshop CS5, turned that mockup
into assets for the game, and then just had to find motivation to
actually code the game.

I used my Image Spectrumizer to convert most of the assets from
Photoshop to the speccy. The moving bits (checkmark and faces) I
converted with a simpler tool that can be found in my speccy github
repo. One awkward consqeuence was that the stuff produced by the
spectrumizer had ink and paper reversed compared to the other stuff
(including the ROM font). I could have spent time fixing that, either
by making a tiny tool to invert everything, or maybe do that at
runtime, but opted to just xor 255 the stuff when drawing to the
screen. It's fast enough.

The line going down and up is done with putpixel. Doing a putpixel on
speccy's weird screen is rather convoluted; first you have to
calculate the address if the byte the pixel exists in (the address is
a bit swizzled: 0 1 0 Y7 Y6 Y2 Y1 Y0 Y5 Y4 Y3 X4 X3 X2 X1 X0), and
then do a read-modify-write on the byte. Oh and there's no barrel
shifter on the z80. I opted to do a 8 byte lookup table. In a stroke
of inspiration I just made the putpixel use xor to put the pixel, so
the same call can be used to undo the effect.

I didn't feel like reinventing a random function so I found one on
the net somewhere. I also struggled with the coordinate calculation
for some reason, so I found a ready function for that as well.
There's two routines for it in the source code now - one that
calculates 8x8 coordinates and one that calculates 1x1 - the 8x8 one
is my original code I wrote for the z80 tutorial.

Code writing wise, I first did everything that did something visual;
every element has its own function. The tick is a 8x8 pixel element,
the bobbing rod is a 8x16 one, the faces are 24x24, and text output
is another one. And since we need numeric displays, that's its own
routime (although it uses the text output routine for actual output).

The input is literally a "any key" routine. It checks if any key on
the speccy is pressed or not. I also considered adding Kempston
support, but if you don't have a Kempston interface, the port may
have random values in it, so I left it out.

For audio, I didn't even consider using some third party bleep
routine, but rolled my own. The audio requirements are fairly
minimal, after all.

While doing the game logic I figured that the game is annoying enough
without having a random chance of not getting a fish, so that was
out. There's depths where you can't find any fish, anyway. Having
fish that only exist in one depth was also super annoying, so that
was out. As a final adjustment I added the "top" and "deep" hints for
the fishes that have depth < 10 and depth > 20.

I used my mackarel to turn the raw binary output from the assembler
into a tap file along with the loading screen and stuff.

I tried to find testers for the game, but only found a couple of
people willing to give it a spin, and the feedback was basically "it
works", so that was that. I found some bugs in my own testing and
fixed those, but never received any other bug reports. The excelent
writeup in csscgc suggests there are bugs, though. Oh well.

You can find the entire z80 assembly source code here on github, and
if you want to take the game for a spin, the game .tap file and
online emulator are here on this site.

Mass Effect Trilogy

February 10th, 2025 (permalink)

So I decided to play through the Mass Effect trilogy again after
playing Andromeda. And I was downright floored just how much better
those games are. Each of the games has way, way more variety than
Andromeda does, even though Andromeda allegedly has more voice lines
than the whole trilogy has.

There's little I have to add to what I said earlier. Still love the
trilogy, still enjoyed Andromeda, still feel the couple novels I've
read were good. I also ordered the comics, which are probably the
last of Mass Effect I'll see. There may be a TV show being produced,
but it's by Amazon - likelihood that I'll watch it is slim. Fifth
Mass Effect game is in preproduction at BioWare / Electronic Arts,
with approximately zero people from the original trilogy team, so I
don't have high hopes for it.

Assuming they manage to make the new game, I fear it will be in the
same risk-averse stuff as Andromeda, which I feel is not a Mass
Effect game, but something else with a Mass Effect coat of paint.
Kinda like Beverly Hills Cop 3..

Anyway, feeling a bit empty now. All those goodbyes. Sigh.

In other news, the new year resolution progress:

  * Math: Asked around, and was referred to OpenStax math books and
    they've been just what I was looking for. Been reading up on
    stuff and it's been interesting: first, I've forgotten so much
    about math! and second, I learned math in Finnish, so re-learning
    it in English is kinda fun. There's also focus on different
    things than how I was taught.
  * Exercise: I've hit the rowing machine on most mornings. I've
    found it to be somewhat meditative to just close my eyes and do
    100 pulls, which happens to result in about 500 meters. Some days
    I do more. I probably should do 2000 meters, but 500 is better
    than nothing.
  * SoLoud update: haven't touched it. SoLoud on next: haven't
    touched it.
  * MuCho on next: played around with the source code a bit, and
    realized it needs quite bit of work, considering the C code reads
    the whole data file into memory, and has 64k memory buffer. The
    next has a 8 bit CPU that only sees 64k (at a time), so this
    probably won't work.

Caffeine Migraine

January 11th, 2025 (permalink)

I have strong suspicion that caffeine causes migraine for me. Let me
explain.

Some years ago I ended up in a situation where we had frequent (and
consistent) visitors for whom we'd make coffee. Since we don't drink
coffee ourselves, pre-ground coffee would go bad, so the most
sensible thing was to buy beans instead. And once you have beans, you
need a proper coffee grinder, and might as well get an Aeropress. And
after all this investment, you kinda have to learn how to make proper
coffee.

So I did. We picked one light roast bean type that was readily
available locally, and I tuned the grind size so that it wasn't too
acidic or too bitter for my taste. Wife was also surprised to find
that there was coffee that she liked.

That lead to a routine where I'd drank one cup of coffee a day. Not
excessive by any means. That went on for maybe a month, and then the
migraine started. I'd stop drinking coffee, migraine went away, tried
a cup, and bang. So it was somehow related to the coffee. I measured
my blood pressure, got bloodwork done and my medication was changed,
but it was still pretty weird that coffee would cause this. Coffee
does rise your blood pressure for a brief time, but the amount I took
shouldn't make a difference. Liquorice also does, and that has never
caused problems for me (that I know of).

Fast forward a couple of years, and instead of a candy advent
calendar I figured I'd try a tea advent calendar instead. Which means
a cup of tea a day. See where I'm going with this?

On day 21, migraine. Day 22, migraine. I stopped drinking tea, no
migraine. Haven't drunk tea since, haven't had an attack since.

This makes me wonder. I've had a cup of coffee or tea occasionally
without any problems, but these two "cup a day" experiments have lead
to migraine attacks. Looking back to when I've randomly gotten an
attack, has caffeine been part of the equation? Perhaps? When I was a
kid and got very severe attacks pretty often, I was drinking a lot
(and I mean a lot) of coca-cola. The attacks ended when I was doing
my military service, so I've attributed it to getting in shape
physically, but.. I also had a lot less caffeine during that time.

Who knows? The best I can do is just avoid routinely drinking
caffeinated things.

Mass Effect Andromeda

January 10th, 2025 (permalink)

Mass Effect: Andromeda came out in 2017, so I think I'm allowed to
drop a few spoilers here, not that they really matter in your
potential future enjoyment of the title.

I loved the original Mass Effect. For the first time in a very long
time, the game gave me what Tim Schafer calls "promise of infinite
possibilities". At the beginning you couldn't "feel" the edges of
your playground. You did find them pretty soon, but still, that was
quite the feat. With the "living the movie"-like feel, great writing,
great music and, for the time, great graphics, it just grasped me.
The fact that it was basically modern-day Star Control 2 (the #1 game
of my formative years) may have played a part.

Then they said that they "compiled a list of player's feedback and
addressed every single one" while making Mass Effect 2. I felt that
was a mistake.

Well, ME2 was much more "game" than an immersive experience. Sure,
many of original ME1 mechanics were, in retrospect, rather clunky,
which was very apparent to me when I re-played the game later on.

ME3 was only available on EA's store so I didn't play it for a
longest time. But I did buy a couple of ME novels by the original
game writers. They're a fun read.

Anyway, ME3 was released with some complaints, especially about the
ending. Apparently someone in the management felt that the original
plans for the plot, which made sense and were good, were "leaked" or
something, so they had to pull something new out of the hat and the
ending was weird. They patched the game later on to make the ending
more sensible (I think?), and the version of ME3 that I eventually
played was the legendary edition, so I never saw the original ending.
ME3 is not a bad game, but it's lesser than the original, or even
ME2.

Then Mass Effect: Andromeda was released, to very negative press. I
largely ignored it. It was on sale for a fiver over the holidays, so
I figured, what the heck.

I've been trying to form some kind of idea of what went wrong with
the game, and figured I'd write a blog post about it.

First off:

  * Did I enjoy the game? Yes.
  * Was it great? Not really.
  * Was it as bad as I was lead to believe? Probably not.

It's.. content.

The first impression of the game comes from the graphics, or more
precisely, faces of the characters. They're.. ugly. It feels like
it's impossible to make a pretty character in the character editor. I
don't know what went wrong there. You kinda get used to it. As space
visuals go - I mean planets and such - it's very pretty.

The writing feels like a fan fiction of Mass Effect. There are some
pretty good or pretty funny bits here and there, but you can't have a
game of this size without hitting the mark sometimes. Since you have
to have some shocking things, there's a religious character in the
game. Not in "scifi religion" but real-world religion. Like most
things in this game, it never goes anywhere. Elsewhere, alien lore is
just dropped on your lap out of nowhere, used in one side mission and
then never referenced again. Some writer apparently fell in love with
"colors we don't have names for", since that phrase stood out several
times.

There's several story beats that apparently had to be in the script
because they were so nice in earlier games. Except they're very
watered down. Case in point:

(and here's the more majorly spolery bit)

In ME1 you have a couple of squad mates who you have been building
relationships with over time, maybe even romantical ones, and at one
point you have to choose which one lives and which one dies. It's
quite dramatic, and comes as a shock. The one who dies will no longer
be there, and your choice has literal consequences. The surviving
character even pops up in ME2 and ME3.

In ME:A you're doing a mission, meet someone during that mission, and
then are asked to either save that character's life or someone else
you haven't met. I haven't played the alternative choice, but there's
a couple of lines of dialogue from someone complaining about your
choice and that's about it.

There's also a "you must disobey the authority to save the day" story
beat in ME:A as in earlier games, which pretty much comes out of
nowhere, and results in no consequences whatsoever.

(end of spoilery bit)

Overall, I don't know if any of my choices changed anything in ME:A,
even though the game claims that they do.

At around 15 hour mark I realized I don't even remember a single
character's name nor do I care about any of them. I mostly remembered
doing 3d platforming and sudoku puzzles.

The game has "only" two new alien races even though you're in a new
galaxy. I'm fine with that, and there's even some story
justifications for it. (The same justification can explain the
identical fauna on all the planets). What does bother me is that the
aliens are not on par with the old ME alien designs.

Technically it's pretty obvious that different worlds were
implemented by separate teams. Some things that should be common just
work slightly differently in different places. Why are doors so slow
on one world? Why is there a text menu to pick where to go, while
everywhere else you'd just use a different door or something? This is
definitely nitpicking, but these would have been super easy to fix.

Another problem with the game is that there's just so much stuff in
it. You assign strike teams to do something that takes real-world
time to do. I don't know why, or what the benefit was. There's
mining. There's cryo pod assignment stuff. There's crafting. There's
research. There's collect-a-thon of a billion different things that
don't matter. There's planetary investigations from orbit (click on
planet, wait for transition, click on planet, maybe find something,
click on planet, move to the next one). The game would have been
better off by trimming all that unnecessary crap. Maybe leave the
(very pretty) planet orbits, flesh them out a bit.

Gameplay wise it felt more like Assassins Creed than Mass Effect.
Pick point of interest on the map, go there, pick something up, press
a button or shoot a guy there, move to the next one. There's no
towers to climb (well.. sort of) but there might as well be. This
sounds like a complaint, and it kinda is, but then again, I enjoy
this kind of gameplay. It's just not what I want from an ME game.

You're driving around a lot. A lot. I actually used a cheat to get
infinite boosts just to make that more tolerable. Didn't use the
cheat for anything else. I also played at easiest difficulty level
and ran through many of the fights the game was setting up. Turns
out, most of them are optional. (Towards the end of the story
missions there are some places where you run past enemies just to
find a door that doesn't open before you go back and shoot them,
which makes zero story sense). Even with these shortcuts the game has
still taken me some 50 hours to play, and I think I'll still go back
to finish a few side missions.

Yes, the game doesn't end at the credits. It's one of those
sandboxes. Which means the story can't change a many aspects of the
world. Nor can the side missions. They're just content that can be
played in whatever order.

As I'm writing this I have ME soundtrack playing, and realized that
in ME:A the soundtrack is.. just there at the background somewhere,
and doesn't make a fuss. That's pretty much what the whole of ME:A
is; it's ME themed, but doesn't want to draw attention to itself.

MMXXV

January 1st, 2025 (permalink)

It's a new year; let's hope it's more positive one than the dark
clouds in the horizon seem to predict.

And here's the new year demo:

And here's a reminder that my Ko-Fi shop exists

I don't have the greatest track record when it comes to new year
resolutions, but here's my sketch for next year:

  * I should dig my old math school books and do a refresh. I've
    found that I've forgotten way too much of it. I'll have numpy/
    sympy to play with, though, so it's not all manual work.
  * I should re-start daily excercise, either cycling, excercise bike
    or rowing machine. Exceptions being when there's yardwork (such
    as snow), or if I'm sick.
  * I should merge a bunch of pull requests and make a new stable
    release of SoLoud, it's been too many years. Maybe also finish
    some kind of package for the ZX Spectrum Next users, as I was
    already 90% there. Maybe it's possible to use the terminal driver
    in BASIC, something I have no idea how it would work.
  * And finally, I should make a first pass of MuCho/dialogtree for
    ZX Spectrum Next, pure C, using z88dk.

Apart from all that, I have this annoying idea buzzing in my head
that I should pick some faraway city, and then obsess about its
history like only someone who has never been there can, and then spin
a game or book or something on a fantasy setting on top of that.
Maybe use all of these story snippets I've played with over the
years. Who knows.

Likelihood that I manage all of the above? About zero. But maybe
something will come out of it.

Next year

  * 25
  * 24
  * 23
  * 22
  * 21
  * 20
  * 19
  * 18
  * 17
  * 16
  * 15
  * 14
  * 13
  * 12
  * 11
  * 10
  * 09
  * 08
  * 07
  * 06
  * 05
  * older

Previous year

Site design & Copyright (c) 2025 Jari Komppa

Possibly modified around: 20 May 2025

(1,0) (1,3) (-2,-1) (3,-1) (3,-3) (-2,-3)