Record field padding for pointer types
lazarus / 62-General

----------------------------------------------------------------------
kupferstecher · 2022-03-14 15:57

Hello,

for communication purposes I need the same binary representation of a
(packed) record on different targets (32/64bit). The problem is that
ansistrings and dynamic arrays have the size of a pointer, which is
32bit on 32bit targets and 64bit on 64bit targets. For the field
itself it wouldn't be an issue, but the alignment of all following
fields will be different. Is there a way to force pointer/ reference
types to a 64-bit alignment to be compatible on different bitnesses?

Below the only solution I came up with, using ifdefs to include
padding for 32bit targets. But thats troublesome and errorprone (easy
to forget), so the question is if there is something easier, like a
directive for the record or a type definition that enforces padding?
Or anything else.

Thanks!


Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Type 
TMyRec = packed record    p1: Int32;    p2: Ansistring; {$IFDEF CPU32}
res1: Pointer; {$ENDIF}    p3: array of uInt32; {$IFDEF CPU32} res2:
Pointer; {$ENDIF}  end;

----------------------------------------------------------------------
Thaddy · 2022-03-14 16:14

{$packrecords 1,2,4,8,16,32} 1, 2, 4, 8, 16, 32, C, NORMAL or DEFAULT
(or 0). Choose the padding you want..
See https://www.freepascal.org/docs-
html/current/prog/progsu60.html#x67-660001.2.60
It is a local directive so can be used on a per record basis. (use
{$push/pop})
If you pack the records size 8 on both platforms you cover both 32 bit
and 64 bit pointers

----------------------------------------------------------------------
kupferstecher · 2022-03-14 16:22

Hi,

the other fields should be packed (byte alligned) to keep the
structure small. Only the pointer-likes should be padded.
Yes, I probably should have mentioned that

----------------------------------------------------------------------
Thaddy · 2022-03-14 16:30

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
{$packrecords 1}Type  TMyRec = packed record    p1: Int32;    p2:
Ansistring;    p3: array of ptrUint;// no padding at all, ptruint (or
nativepointer) is 32 bit on 32 bit and 64 bit on 64 bit  end;Otherwise
just use a variant record to pad the 32 bit to 64 bit. No ifdefs
necessary. Note p2 Also needs padding...

----------------------------------------------------------------------
kupferstecher · 2022-03-14 16:56

A variant record would work, but seems even more complicated than the
ifdefs.

----------------------------------------------------------------------
mas steindorff · 2022-03-14 18:48

what I would do is to define a separate record (overlay/union) that
included both sized pointers and maybe include a 64bit int as well. 
it's size will always be 64 bit. 
then use it to replace your pointers in your TMyRec.  then create a
single define or type cast that referenced the correct pointer based
on the target
Use that "define" or type cast to access the pointer in your code.
your record keeps it size constant for both 32 and 64 bit targets.

----------------------------------------------------------------------
MarkMLl · 2022-03-14 19:12

I am gravely concerned by any paragraph which contains both "pointer"
and "communications".

Whatever you do, make damn sure that your code tests all sizing
assumptions at startup: preferably by something that can't be disabled
since a small amount of size or performance overhead is a negligible
penalty when compared with the sort of things that could go wrong.

MarkMLl

----------------------------------------------------------------------
kupferstecher · 2022-03-14 20:33

The background of my question can be read here:
https://forum.lazarus.freepascal.org/index.php/topic,58574.0.html

in short: For a TCP communication I want to apply a record to a byte
array (the TCP message) via pointer. The record contains the data I
want to communicate. For basic types with fixed length it works well,
but the dynamic types (ansistring, arrays) have the data on the heap.
I already wrote a procedure using RTTI moving the heap part into the
message directly behind the record. The reference of the dynamic type
to the heap data is then replaced with the offset value of the data
written in the message. On the receiver side this is reversed, namely
the heap data for the string or array is allocated and the data moved,
so the record fields can be easily accessed in the receiver function
just using the record fields. So sender and receiver side need to have
the exact same binary representation. And the bitness of the
"pointers"/references interfere here. Simple padding would be enough,
the more simple the better.

Quote from: MarkMLl on March 14, 2022, 08:12:17 pmWhatever you do,
make damn sure that your code tests all sizing assumptions at startup:
preferably by something that can't be disabled since a small amount of
size or performance overhead is a negligible penalty when compared
with the sort of things that could go wrong.
Yes, would be good, but how? The best would be to have only one place
for the record definition without distinguishing the bitness. But from
the answers now it seems there is no such feature.

----------------------------------------------------------------------
mas steindorff · 2022-03-14 23:15

for my TCIP stuff, I've always defined a byte as either a rev or a
record type const that the Rx then uses to decode the rest of the byte
fields. But as MarkMLI said, pointers are not passed and I have not
tried to do so.

----------------------------------------------------------------------
MarkMLl · 2022-03-15 08:05

Quote from: kupferstecher on March 14, 2022, 09:33:46 pmin short: For
a TCP communication I want to apply a record to a byte array (the TCP
message) via pointer. The record contains the data I want to
communicate. For basic types with fixed length it works well, but the
dynamic types (ansistring, arrays) have the data on the heap.

I think it might be useful to change a couple of bits of terminology.
The key here is that you are trying to /serialise/ (and later
/deserialise/) data, and since you are sending binary data which
cannot be explicitly terminated you need to embed /offsets/ and/or
/lengths/ in the message.

Serialisation and deserialisation have been dealt with many times
before, and these obviously include consideration of fetching and
storing your data: your program should not be trying to go to the heap
etc. to get the bytes representing a character sequence but instead
relying on the language facilities to manipulate a string.

So the problem boils down to four things:

a) Knowing the encoding and locale of a string.

b) Knowing the size and endianness of an integer or real.

c) Qualifying these with a count if necessary to represent there being
multiple associated elements (i.e. an array in Pascal terms).

d) Choosing a format to embed the offset of the start of each field
etc.

You might also need to consider what each field represents, i.e. the
schema of the data being transmitted. At the very least, you will need
to start the message with a magic number which you change if any
detail which is not described internally is altered.

From the above you will obviously see that it is non-trivial. This
sort of thing has been tackled in the past by tagged/chunked formats
such as TIFF and AIFF, but these days it's far more common to
serialize files as zipped XML, and something similar could potentially
be used for communications.

MarkMLl

----------------------------------------------------------------------
Thaddy · 2022-03-15 08:13

Or JSON, which I prefer over XML. If speed is een issue one can
serialize the record into an array with #0 as separators and #00#00 as
terminator. FPC has convenient helpers for that, like
ArrayStringToPPChar, where  the Payload is PPChar^, which is written
sequentially. There are more such gems.
Point is that pointers are not the way to go with communications as is
already well explained. You need to serialize the actual data. That's
not too difficult as opposed some suggestions.

----------------------------------------------------------------------
PascalDragon · 2022-03-15 08:13

Quote from: kupferstecher on March 14, 2022, 04:57:47 pmfor
communication purposes I need the same binary representation of a
(packed) record on different targets (32/64bit). The problem is that
ansistrings and dynamic arrays have the size of a pointer, which is
32bit on 32bit targets and 64bit on 64bit targets.
That makes no sense. If you have a dynamic array or a AnsiString as
field you can't use its binary representation(*), because the array
and string data will be anywhere but the record. Thus there is no real
need to pack this record. Any record you send “on the wire” or store
in a file shouldn't use pointers, but instead offsets that you fill in
when serializing the record. Then you can also use a representation
that's the same no matter the bitness. Essentially you need two
records: one for the serialized representation and one with which you
work and then you convert from one to the other when (de)serializing.

(*) In fairness one can, but one shouldn't, especially if one doesn't
know exactly how these types work which - no offense - you don't,
judging by your other question regarding managed types. Not to mention
that the metadata of these managed types might be different between
compiler versions, cause they are more or less an implementation
detail.

----------------------------------------------------------------------
kupferstecher · 2022-03-15 12:23

Quote from: mas steindorff on March 15, 2022, 12:15:21 amdefined a
byte [...] that the Rx then uses to decode the rest of the byte
fields. But as MarkMLI said, pointers are not passed and I have not
tried to do so.
Actually thats how I want to do it. And I don't pass the pointer at
all. Obviously it doesn't make sense to pass it, but the following
fields still have to be well aligned. I do 'abuse' the pointer field
as offset value, but thats the easy part and doesn't need 64 bit.

Quote from: MarkMLl on March 15, 2022, 09:05:08 amSerialisation and
deserialisation have been dealt with many times before And I wonder,
why its not done the way I try to. Or put in other way, why there
isn't better language support for that. Especially as in the older
times binary files were more common.

Quote from: MarkMLl on March 15, 2022, 09:05:08 amSo the problem boils
down to four things:
a) Knowing the encoding and locale of a string.
b) Knowing the size and endianness of an integer or real.
c) Qualifying these with a count if necessary to represent there being
multiple associated elements (i.e. an array in Pascal terms).
d) Choosing a format to embed the offset of the start of each field
etc.
a) UTF-8
b) Little Endian. That could be an future issue, by now Win/Linux on
intel or ARM are all Little Endian, I think. And even if the target is
Big Endian, the order could be reversed while serialization.
c) This is given by the length information. The element size is part
of the array definition.
d) The offset of each field is given by the record, only the offsets
of the data part of strings and arrays I have to store seperately.


Quote from: Thaddy on March 15, 2022, 09:13:11 amOr JSON, which I
prefer over XML.I still do consider that, but I first want to fully
understand what is possible in freepascal using the binary method.

Quote from: PascalDragon on March 15, 2022, 09:13:43 amThat makes no
sense. If you have a dynamic array or a AnsiString as field you can't
use its binary representation(*), because the array and string data
will be anywhere but the record.
Now I'm not sure if the concept I have is really understood (its quite
difficult to explain). The full binary representation isn't used, its
only the characters of the AnsiString that is serialized.

And I wonder, that no one sees the simplicity in usage of that
solution? The following code is the complete definition of a command.
All magic is in the Serialize method using the type information of the
record, so it is universally translating different record structures.
And everything with the speed benefit of binary data.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Type 
TCmdHello = packed record    TimeStamp: uInt32;    Caption:
Ansistring; {$IFDEF CPU32} padding: Int32; {$ENDIF}  end; Const   
cCmdHello= 1234;  Procedure SendHello;var  msg: TTCPMsg;  params:
^TCmdHello;begin   msg:= TTCPMsg.Create(cCmdHello,SizeOf(TCmdHello)); 
params:= msg.DataAddress;   //Result:= @buffer[0]   
params^.TimeStamp:= 12;  params^.Caption:= 'SayHello';         
msg.Serialize(TypeInfo(TCmdHello));  msg.Send;   end;
Quote from: PascalDragon on March 15, 2022, 09:13:43 amEssentially you
need two records: one for the serialized representation and one with
which you work and then you convert from one to the other when
(de)serializing.Thats what I want to avoid. And I'd face the same
problem of dynamic data that can't be represented in the record for
the serialized representation.

Quote from: PascalDragon on March 15, 2022, 09:13:43 amNot to mention
that the metadata of these managed types might be different between
compiler versions, cause they are more or less an implementation
detail.
That is really an issue and especially the deserialization of a
dynamic array seems like a hack, as the heap allocation has to be done
using type information and thus there is not the actual type that
could be passed to SetLength. But to be clear, the "network data"
binary representation is well defined, so if the internal
representation really changes in the future, then I still could
"manually" reformat the data. Actually the Serialize procedure even
could convert the record for sending to JSON.

----------------------------------------------------------------------
kupferstecher · 2022-03-15 12:30

And for the sake of completeness also the receiver side:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Procedure
RecvHello(aMsg: TTCPMsg);var  params: ^TCmdHello;begin 
msg.Deserialize(TypeInfo(TCmdHello));  params:= msg.DataAddress;   
writeln('TimeStamp:',params^.TimeStamp);  writeln('Caption:',
 params^.Caption);    Finalize(params^);  aMsg.Free; end;

----------------------------------------------------------------------
PascalDragon · 2022-03-15 12:52

Quote from: kupferstecher on March 15, 2022, 01:23:59 pmAnd I wonder,
that no one sees the simplicity in usage of that solution? The
following code is the complete definition of a command. All magic is
in the Serialize method using the type information of the record, so
it is universally translating different record structures. And
everything with the speed benefit of binary data.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Type 
TCmdHello = packed record    TimeStamp: uInt32;    Caption:
Ansistring; {$IFDEF CPU32} padding: Int32; {$ENDIF}  end; Const   
cCmdHello= 1234;  Procedure SendHello;var  msg: TTCPMsg;  params:
^TCmdHello;begin   msg:= TTCPMsg.Create(cCmdHello,SizeOf(TCmdHello)); 
params:= msg.DataAddress;   //Result:= @buffer[0]   
params^.TimeStamp:= 12;  params^.Caption:= 'SayHello';         
msg.Serialize(TypeInfo(TCmdHello));  msg.Send;   end;
I agree that it is quite elegant, however I would have removed the
need for packed by having the Serialize simply write the data without
any padding and deserializing it without expecting any padding (and
transparently converting strings and arrays) as you need the field
offsets only for reading the record for serializing and writing the
record for deserializing, but not for the “on wire” format. You then
of course need to use types that are equal on different bitnesses for
counts and such, but you already did that with UInt32 in your example.

----------------------------------------------------------------------
avra · 2022-03-15 13:13

Quote from: kupferstecher on March 15, 2022, 01:23:59 pmQuote from:
Thaddy on March 15, 2022, 09:13:11 amOr JSON, which I prefer over
XML.I still do consider that, but I first want to fully understand
what is possible in freepascal using the binary method.
How about https://msgpack.org or https://github.com/lalexs75/protobuf-
fpc ?

----------------------------------------------------------------------
kupferstecher · 2022-03-15 14:33

Quote from: PascalDragon on March 15, 2022, 01:52:52 pmhowever I would
have removed the need for packed by having the Serialize simply write
the data without any padding and deserializing it without expecting
any padding (and transparently converting strings and arrays) as you
need the field offsets only for reading the record for serializing and
writing the record for deserializing, but not for the “on wire”
format.Yes, it would be definitely more robust and more "user
friendly". But it would involve a lot of copying and scanning for each
type, which have to be dealt with seperately. With the packed record
the Serializer only needs to consider dynamic data types, the fixed
ones are already in place. (This means there could be even records in
the record as long as they only contain fixed size variables. On the
other hand they probably gain no use case). I am aware that extending
the message buffer for the dynamic part also involves a copy, which
should be quite fast, though, and is only on the sender side.

Hm, I have to think about it.

Quote from: avra on March 15, 2022, 02:13:03 pmHow about
https://msgpack.org or https://github.com/lalexs75/protobuf-fpc ? That
is interesting as well! It should be no disadvantage to use some
standard

----------------------------------------------------------------------
SymbolicFrank · 2022-03-15 14:37

JSON is simpler, but it doesn't hold meta-data (perhaps null, but
that's it). So all your ints end up int64, because you don't know the
size. Etc. If you want limits, XML is better.

The first time serializing/deserializing is hard, the tenth time it's
easy.

----------------------------------------------------------------------
Thaddy · 2022-03-15 14:45

JSON can hold meta data but you have to provide it. XML is a bandwidth
hog. Both are easy, though.

----------------------------------------------------------------------
d.ioannidis · 2022-03-15 15:20

Hi,

 maybe I'm wrong, but, it seems to me that, you want something like
ASN.1 with tools like ASN1SCC explained in useful old technologies.

regards,

----------------------------------------------------------------------
MarkMLl · 2022-03-15 15:29

Quote from: Thaddy on March 15, 2022, 03:45:22 pmXML is a bandwidth
hog.

Which is why I specified that it should be zipped. However I think
there is a problem which applies to any zipped or gzipped stream: it's
very difficult to seek to a specific point in the serialized data, and
even if an external index has been constructed it is very difficult to
resynchronise having sought.

So there's a lot to be said for using a sequence of binary records,
but they need at least one level of metadata and the data stream
should either contain indexing information or be compatible with an
index built on the fly: in short, it should be "database-like".

MarkMLl

----------------------------------------------------------------------
Thaddy · 2022-03-15 15:31

msgpack functionality can be easily integrated in fcl-json, it seems.
For easy implementation, needs 3.2.0 or higher.

----------------------------------------------------------------------
PascalDragon · 2022-03-16 07:44

Quote from: kupferstecher on March 15, 2022, 03:33:51 pmQuote from:
PascalDragon on March 15, 2022, 01:52:52 pmhowever I would have
removed the need for packed by having the Serialize simply write the
data without any padding and deserializing it without expecting any
padding (and transparently converting strings and arrays) as you need
the field offsets only for reading the record for serializing and
writing the record for deserializing, but not for the “on wire”
format.Yes, it would be definitely more robust and more "user
friendly". But it would involve a lot of copying and scanning for each
type, which have to be dealt with seperately. With the packed record
the Serializer only needs to consider dynamic data types, the fixed
ones are already in place. (This means there could be even records in
the record as long as they only contain fixed size variables. On the
other hand they probably gain no use case). I am aware that extending
the message buffer for the dynamic part also involves a copy, which
should be quite fast, though, and is only on the sender side.
You know the sizes from the type information, so whether you index by
offset or “sum of sizes of fields encountered” isn't that much
difference.

----------------------------------------------------------------------
kupferstecher · 2022-03-16 11:32

Quote from: PascalDragon on March 16, 2022, 08:44:15 amYou know the
sizes from the type information, so whether you index by offset or
“sum of sizes of fields encountered” isn't that much difference.
The difference is that I have to move all field values to the correct
place calculated by the “sum of sizes of fields encountered”. While
the fields by offsets are already in the correct place.
But I agree, in total yours is the better solution.

Quote from: Dimitrios Chr. Ioannidis on March 15, 2022, 04:20:24
pmASN.1
ASN1SCC
useful old technologies
These are very interesting links, thank you!

---
Also thanks to all others!

I guess I just have to start, make my mistakes, then remember your
advices and choose the suitable way later 

Here the latest version:
https://forum.lazarus.freepascal.org/index.php/topic,58574.msg437447.h
tml#msg437447

----------------------------------------------------------------------
MarkMLl · 2022-03-16 12:07

Quote from: kupferstecher on March 16, 2022, 12:32:47 pmI guess I just
have to start, make my mistakes, then remember your advices and choose
the suitable way later 

Leaving aside all other considerations for the moment,   BE TOLD: put
something in your comms protocol (or file format etc.) so that it
becomes immediately apparent if two programs are using slightly
different versions of the the protocol/format either because one has
just been recompiled but the other hasn't (or has, but hasn't been
restarted) or because you're working on two different systems with
different word size or endianness.

If you don't, something unexpected will happen and you will eventually
waste a lot of time chasing an unexpected binary format change... and
I can assure you that I speak from experience. To reinforce that with
others' experience, you might also like to consider
https://en.wikipedia.org/wiki/Portable_Network_Graphics#File_header
where the first four bytes should be written and read as an integer
specifically to detect endianness, size and numeric representation
issues. You might also like to consider protocols such as Kermit that
were specifically designed to accommodate non-transparent comms links.

MarkMLl

----------------------------------------------------------------------
kupferstecher · 2022-03-16 13:38

Quote from: MarkMLl on March 16, 2022, 01:07:51 pmBE TOLD: put
something in your comms protocol (or file format etc.) so that it
becomes immediately apparent if two programs are using slightly
different versions of the the protocol/format either because one has
just been recompiled but the other hasn't (or has, but hasn't been
restarted) or because you're working on two different systems with
different word size or endianness.
Yes, Its good advice.

The endianness I can just check:
{$IFDEF ENDIAN_BIG} Raise Exception.Create('Serializer is not
implemented for Big endian systems');

The word size matters for the string and array fields, I could check
in a 32bit ifdef in the Serializer if its followed by a padding.

For the changed record structure I'm not sure now. Perhaps a request
of the protocol version when starting the connection could be enough.

----------------------------------------------------------------------
MarkMLl · 2022-03-16 13:49

Quote from: kupferstecher on March 16, 2022, 02:38:19 pmThe endianness
I can just check:

No: you need to make those checks at runtime, otherwise you will
sooner or later have problems.

Records (and other data structures) you can check by documenting the
size of each one and using assertions to always check that the
compiler is allocating just that number of bytes: put it in the
initialisation block of each relevant unit.

A few weeks ago I had to use a Linux kernel API where some damnfool
had defined a data structure with three integers followed by a pointer
which the C compiler (GCC?) had quietly packed.

Look, I'm very much old-school as far as forums and things are
concerned: I can't be bothered to memorise any more formatting tags
than necessary and I /never/ use animated icons. But I took the
trouble to use one for you...

MarkMLl

----------------------------------------------------------------------
kupferstecher · 2022-03-17 09:50

Quote from: MarkMLl on March 16, 2022, 02:49:20 pmand I /never/ use
animated icons. But I took the trouble to use one for you...And I do
feel honored