* * * * *
                                        
   The AI has come for my code part II, Monty Python's Flying Circus Edition
                                        
It's baaaaaaaaaaaaaack [1]!

Yes, I replied to the previous PR (Pull Request) with “Next time, submit the
data that will cause the memory overright. Until then, this PR is closed.” It
replied with the usual obsequious apology and said that it overstated the
severity (“CRITICAL” in all caps) of the bug. And I thought that would be the
end of that, that it would go off and spend tons of tokens trying generate
input that would attempt to prove the bug and not be able to do it.

But no, it found a novel approach. At first, I was furious, wanting to nuke
the LLM from orbit (that's the only way to make sure), but the pure silliness
of what it did is just proved too amusing to me.

This time, it managed to “patch” all four calls to memcpy(). Basically, it
changed the code from:

-----[ C ]-----
opd->sz = min( ... , sizeof(opd->bytes));
-----[ END OF LINE ]-----

to

-----[ C ]-----
opd->sz = (uint16_t)min( ... , sizeof(opd->bytes));
assert(opd->sz < sizeof(opd->bytes));
-----[ END OF LINE ]-----

Okay, fair enough. opd->sz is of type uint16_t and min() returns a size_t, so
the cast is being explicit about that. Also the assert() is just checking
that the result of min() is correct, but min() is also a trivial function:

-----[ C ]-----
static inline size_t min(size_t a,size_t b)
{
  return a < b ? a : b;
}
-----[ END OF LINE ]-----

It's so trivial that I didn't even think of calling assert(). And sizeof(opd-
>bytes) will easily fit in a unit16_t and it's not like the cast is required
(and I tend to avoid casts unless absolutely necessary, and often times when
it's absolutely necessary, it's a sign the design is wrong). So no, the
patches aren't something I would be interested in. But the code patches
weren't the amusing part No, that would be the “tests” it wrote “proving” the
code was correct.

It wrote, and as Dave Barry likes to say: “I am not making this up,” a Python
implementation of the the C code to prove the C code is correct.

No, it did not write Python code to generate a test file for the assembler,
nor did it write Python code to feed test data directly into the assembler.
No, it transcribed the C code into Python [2] and called it a day.

My God! What have we done?

While I have the temptation to write it back saying it should delete itself,
I'm also curious as to how far it will go in trying to get a PR accepted. How
much money can I get OrbisAI Security [3] to spend trying to “fix” my poor
little assembler? Let's see how deep this Python hole goes … 

[1] https://github.com/spc476/a09/pull/6
[2] https://github.com/orbisai0security/a09/blob/fix-opcodes-memcpy-bounds-check-v001/tests/test_invariant_opcodes.py
[3] https://orbisappsec.com/

Email Sean Conner at sean@conman.org

.