[SOLVED] Indy 10 sends more data than expected?
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
aidv · 2015-09-27 14:32

Ok so last couple of weeks I've been on here trying to get some help
regarding my project.

I've figured most of it out, but some things remain to be solved.

I use Indy 10 TCP for my communication as I find it to be very
reliable, which is good, however I noticed that when I send a
TMemoryStream with the size of 47 bytes, the receiver (and XCode
project for iOS) receives 56 bytes.

And when I send a memory stream of lets say some 100 thousand bytes,
the difference might be up to several tens of thousands of bytes.

What is going on here?

----------------------------------------------------------------------
marcov · 2015-09-27 14:36

How do you measure this? How does the extra data look like?

----------------------------------------------------------------------
aidv · 2015-09-27 14:48

I measure it simply by logging the size of the TMemoryStream to be
sent and the length of the NSData that has been received.

I can't really know what the extra data looks like as it's difficult
to check it with large data.

For smaller TMemoryStreams I don't really have any problems, but with
the very large data, it becomes problematic.

I end all my data "packets" with a specific string looking like this:
<packetEnd>

and when I extract a substring of my total data packet the very last
part of the packet is actually ending with <packetEnd>

and when I extract all data from index 0 up until the beginning of my
file-data, everything is OK as well, starting with <packetStart> +
some extra human-readable text and finally a ':' character followed by
the file-data.

so everything seems to be OK, but still, too much data in it
somewhere.

I have checked the size of the file on the server, and all attributes
are correct.

So I don't really know whats going on here.

EDIT:

Here is my sending code:

Code: [Select]procedure THoster.SendData(aData: TMemoryStream;
AAddress: string);
  var i, u: integer;
      sendIndex: word;
      idb: TIdBytes;
      ctx : TIdContext;
begin
  aData.Position := 0;
  ReadTIdBytesFromStream(aData,idb,aData.Size,0);
  with fTCP.Contexts.LockList do
  begin
    for i := 0 to Count -1 do
    begin
      ctx := TIdContext(Items[i]);
      if AAddress <> '' then
      begin
        if ctx.Binding.PeerIP = AAddress then
        begin
          ctx.Connection.IOHandler.Write(idb, aData.Size, 0);
          break;
        end;
      end
      else ctx.Connection.IOHandler.Write(idb, aData.Size, 0);
    end;
  end;
end;

----------------------------------------------------------------------
aidv · 2015-09-27 15:22

Ok I think I solved the problem by adding the following line to my
code:

Code: [Select]ctx.Connection.IOHandler.WriteBufferFlush;
Now my source looks like this:

Code: [Select]procedure THoster.SendData(aData: TMemoryStream;
AAddress: string);
  var i, u: integer;
      sendIndex: word;
      idb: TIdBytes;
      ctx : TIdContext;
begin
  aData.Position := 0;
  ReadTIdBytesFromStream(aData,idb,aData.Size,0);
  with fTCP.Contexts.LockList do
  begin
    for i := 0 to Count -1 do
    begin
      ctx := TIdContext(Items[i]);
      ctx.Connection.IOHandler.WriteBufferFlush;
      if AAddress <> '' then
      begin
        if ctx.Binding.PeerIP = AAddress then
        begin
          ctx.Connection.IOHandler.Write(idb, aData.Size, 0);
          break;
        end;
      end
      else ctx.Connection.IOHandler.Write(idb, aData.Size, 0);
    end;
  end;
end;
Now the size is correct on both sides, at least for small packets.
Hopefully this solves my entire problem.

----------------------------------------------------------------------
derek.john.evans · 2015-09-27 15:27

Just a couple of things I see:

Why are you reading the stream into a TIdBytes? Why not just write the
stream directly?


I also dont see a  Contexts.UnLockList;  // <<<< Important.

----------------------------------------------------------------------
aidv · 2015-09-27 15:36

Why hello Geepster.

How do I do that? The method asks for a TIdBytes and I found an
example on the interwebs where TIdBytes was used, so I assumed that
that was the correct way to do it?

You ar right, I forgot to add UnlockList.

And I see now that for very large data, the problem remains.

And oh I just noticed that it seems like parts of the next packet in
line to be sent seems to be part of the current packet.

I guess I have the same problem that I had a while back.

----------------------------------------------------------------------
rvk · 2015-09-27 15:43

Quote from: aidv on September 27, 2015, 05:36:39 pmAnd oh I just
noticed that it seems like parts of the next packet in line to be sent
seems to be part of the current packet.
You don't send any size with your data. How does the receiving end
know when the data is complete?? Does it have a timeout? Do you close
the connection? Did you use a buffering-option somewhere?

For a buffering problem see here:
http://stackoverflow.com/questions/25438309/indy10-idtcpclients-
iohandler-send-additional-data
(where you should use WriteBufferClose but I'm not sure you need
buffering at all, see the end of the answer there)

----------------------------------------------------------------------
aidv · 2015-09-27 15:51

I do send size with my data, it's right after <packetStart>.

I think that I need to make a packet queue, so when I send a new
packet, it's put on hold until the client sends a "ready" signal, and
then server flushes everything and and sends the next packet.

Is that a good approach?

----------------------------------------------------------------------
derek.john.evans · 2015-09-27 15:55

I'm a little confused by your design. Are you sending files from a
server to multiple clients?

Is that what you were aiming for? I thought you were sending from
clients to a server?

TIdIOHandlerSocket (which is available for both clients & servers) has
a bunch of Writers including:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
WriteFile(const AFile: String; AEnableTransferFile: Boolean = False):
Int64; override;   and
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
Write(AStream: TStream; ASize: TIdStreamSize = 0; AWriteByteCount:
Boolean = False); overload; virtual; 
The latter is the one Im talking about, which has AWriteByteCount (ie:
sending the size)

But, anyway, by iterating the contexts, your code looks more like a
broadcaster.

----------------------------------------------------------------------
aidv · 2015-09-27 16:04

Well, yes.

My server can receive data from multiple clients, but also send data
to multiple clients, if needed that is.

It is not necessary, but it can.

Oh I see, but I have to ask,
is then sending of the size something that is Indy specific, or is it
part of the TCP protocol?

and finally: it can be a broadcaster, if the parameter AAddress is
left empty, or else it will try to send to the specified address.

----------------------------------------------------------------------
rvk · 2015-09-27 16:10

Quote from: aidv on September 27, 2015, 05:51:55 pmI do send size with
my data, it's right after <packetStart>.
Yes... but that's not used by the receiving program (is my guess). Or
is your receiving program checking the size during the communication?

I thought the third parameter of your Write() should be a boolean
(true/false). If true it sends the size of the stream before sending
the actual data. If the receiving end uses a similar function to first
check the size it needs to receive and then actually receiving that
exact number of bytes it's all done automatically.

The sending of the size of the stream is indy/program specific. But if
wouldn't be hard to program into the server that you first need to
receive the size of a stream and after that the actual stream. The
server can complete the package after receiving the exact number of
bytes.

Quote from: aidv on September 27, 2015, 05:51:55 pmI think that I need
to make a packet queue, so when I send a new packet, it's put on hold
until the client sends a "ready" signal, and then server flushes
everything and and sends the next packet.
Did you break up that file you want to send (of hundred MBs) into
multiple streams already? How large are those packets/streams? And I
don't think you need to send the size of the data in data
<packet/start/end>. If your packets are pre-defined (lets say 1MB per
packet) the server can just check that 1MB against what it actually
received. I thought you wanted to check the validity of the actual
received data so you need to send the MD5 (or hash) with the data. But
you could do that separately after (or before) sending the package.

----------------------------------------------------------------------
derek.john.evans · 2015-09-27 16:13

Quote from: aidv on September 27, 2015, 06:04:30 pmOh I see, but I
have to ask,
is then sending of the size something that is Indy specific, or is it
part of the TCP protocol?

It is implemented by indy as a basic package/chunk transfer. If all
your data is stored as TStream's, then, its easy to send files using
TFileStream, strings using TStringStream, and binary data using
TMemoryStream.

Im unsure how you have your chunks sent now, but, a simple binary
integer is the way to go, which is what indy can do for TStreams.

Using terminators for chunks is a bad idea, (if thats what you are
doing).

----------------------------------------------------------------------
derek.john.evans · 2015-09-27 16:19

Quote from: rvk on September 27, 2015, 06:10:16 pmThe sending of the
size of the stream is indy/program specific. But if wouldn't be hard
to program into the server that you first need to receive the size of
a stream and after that the actual stream. The server can complete the
package after receiving the exact number of bytes.

Hi, rvk. Yep, Indy already has a paired Write/Read for chunked
TStreams.
Code: [Select]procedure TIdIOHandler.ReadStream(AStream: TStream;
AByteCount: TIdStreamSize;
  AReadUntilDisconnect: Boolean);   

If AByteCount = cSizeUnknown (ie: -1), then a byte size is read from
the connection.

----------------------------------------------------------------------
aidv · 2015-09-27 16:34

rvk:
The size is used to check if the length of the data is correct.

The size of the packet can be very large, but also very small.

Right now the packets I am testing with are around 300kb in size, just
for a test.

And the packets contain all essential information, such as
start/endsequences, size of the packet, size of data.

Geepster:
What is a terminator? You mean like my <packetEnd> string at the end
of my chunks?

Actually, if a packet is very large, it is automatically sent in
several chunks, and I am utilizing that.

Ofcourse if I need to send several MB of data, then I split that data
in to independent chunks and then send each chunk as a packet.

----------------------------------------------------------------------
derek.john.evans · 2015-09-27 16:45

Quote from: aidv on September 27, 2015, 06:34:36 pmGeepster:
What is a terminator? You mean like my <packetEnd> string at the end
of my chunks?
Actually, if a packet is very large, it is automatically sent in
several chunks, and I am utilizing that.

Yep. It isn't clear from your SendData code whether you send the size
of the TStream or not. If you are, then cool. But, you also said:
QuoteI end all my data "packets" with a specific string looking like
this: <packetEnd>

I, personally don't like packet terminators which require you to read
data, and search for a string, then chop out the chunk, and then
continue reading from the end of the last chunk to get the next chunk.

I know some people love that kinda stuff, but, it just does my head
in.

----------------------------------------------------------------------
aidv · 2015-09-27 17:35

Yes I am actually.

But it's so weird because I send a packet with start sequence plus
some parameters which are a string of  78 bytes in length and a small
file which has the size 119829 bytes an end sequence which is 10 bytes
long, so in total 119917 bytes.

When received the data (in chunks as handled by indy automatically)
the full received data is 176564 bytes.

That means that there is 56735 bytes extra somewhere.

I make sure that the fully received data has its start sequence at the
very beginning, which is true, I've measured it, and also the end
sequence at the very end of the data, which is true as well.

I have also made sure that the start and end sequence occur only once
in the data, which is true (as of right now I've programmed the
software to properly send one packet at a time even if sever sends
have been made, using a send-queue).

So to sum it up:

1. Start and end sequences occur only once in the received data, and
they occur in their respective positions.

2. The length of the pre-data up until the file-data have the correct
size

4. Large packets seem to have a lot of extra data in them.

So my questions now are:

1. Does TMemoryStream add extra data when calling WriteBuffer? (I
append data to the final TMemoryStream which then is sent via Indy)

2. Does indy add data when sending buffer chunks?

3. Does indy encode data being sent as TMemoryStream?

EDIT:

My suspicions goes towards Indys Write method.
I remember that with LNet I had to manually program my software to
send data that was left inside the buffer, maybe Indy sends extra data
when it is sending remaining data from the buffer?

If yes, how is it turned off?

----------------------------------------------------------------------
aidv · 2015-09-27 18:04

I'm an idiot!!!!!

I there's absolutely nothing wrong with Indy 10.

I was doing some weird ass bytes-to-UTF8 conversion in my code, I did
it yesterday and forgot to remove it when I moved over to raw-bytes
processing.

Sorry for wasting your time guys.

Everything works well now. I manage to send data properly and read
data properly now.

Thank you for your help.