Bug in TReadBufStream and proposed fix
lazarus / 64-FPC development

----------------------------------------------------------------------
glorfin · 2017-08-01 14:44

I found that TReadBufStream.Seek function did not work properly,
especially when seeking backwards.
Indeed, here is its code:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TReadBufStream.Seek(const Offset: Int64; Origin: TSeekOrigin):
Int64;begin  FakeSeekForward(Offset,Origin,FTotalPos); 
Result:=FTotalPos; // Pos updated by fake readend; First, and
important. procedure TStream.FakeSeekForward does not seek backward
(as the name suggests).

Second, updated after connet of howardpc:
Third parameter in FakeSeekForward has const modifier, so it should
not be changed by the call.
Then what means comment "Position updated by fake read"?

My suggestion is to rewrite Seek function as follows (in my project
this worked nicely):

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TReadBufStream.Seek(const Offset: Int64; Origin: TSeekOrigin):
Int64;var  NewTotalPos, Delta, NewBufPos: Int64;begin  case Origin of 
  soCurrent:      NewTotalPos := FTotalPos + Offset;    soBeginning:
begin      if Offset < 0 then        InvalidSeek;      NewTotalPos :=
Offset;    end;    soEnd: begin      if Offset > 0 then       
InvalidSeek;      NewTotalPos := Size + Offset;    end;  end;  if
NewTotalPos > Size then    NewTotalPos := Size;  if NewTotalPos < 0
then    NewTotalPos := 0;  Delta := NewTotalPos - FTotalPos; 
NewBufPos := Delta + FBufPos;  if (NewBufPos > 0) and (NewBufPos <
FBufSize) then  begin    FBufPos := NewBufPos;  //if Seek function
moves inside the data which are in the Buffer, we only reposition
FBufPos    FTotalPos := NewTotalPos;  end  else // otherwise find
desired place in the source stream and invalidate buffer  begin   
FTotalPos := FSource.Seek(Offset,Origin);    FBufPos := 0;    FBufSize
:= 0;  end;  Result := FTotalPos;end; 
By the way, what this function must return? I, as mention in comment,
return new position relative to beginning. Is it expected?

Best regards to all!

----------------------------------------------------------------------
howardpc · 2017-08-01 15:47

Quote from: glorfin on August 01, 2017, 04:44:05 pmCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TReadBufStream.Seek(const Offset: Int64; Origin: TSeekOrigin):
Int64;begin  FakeSeekForward(Offset,Origin,FTotalPos); 
Result:=FTotalPos; // Pos updated by fake readend; ... Origin
parameter has const modifier, so it should not be changed by the call.

I don't know where this code comes from, but the Seek call does not
change Origin, it simply plugs its value into the call to Fake. If it
did change its value, and it is indeed a const parameter, the compiler
would refuse the construction.

----------------------------------------------------------------------
glorfin · 2017-08-01 16:16

Oops, sorry, in original post I meant third parameter, FTotalPos. It
is supposed to be changed by the call (see the comment), but in fact
it is not.
But this was side note; important is that you could not move backwards
with this seek function. That is why I had to rewrite it.

----------------------------------------------------------------------
Remy Lebeau · 2017-08-01 23:40

Quote from: glorfin on August 01, 2017, 04:44:05 pmBy the way, what
this function must return? I, as mention in comment, return new
position relative to beginning. Is it expected?

Yes, Seek() is meant to return the new absolute position from the
beginning of the stream.

----------------------------------------------------------------------
glorfin · 2017-08-02 08:53

Quote from: Remy Lebeau on August 02, 2017, 01:40:37 amYes, Seek() is
meant to return the new absolute position from the beginning of the
stream.
Thank you!
Now would be nice to incorporate this new code into BufStream, because
old one really does not work.

----------------------------------------------------------------------
Thaddy · 2017-08-02 09:01

Streams are streams... They are intended to stream in 1 (one)
direction. They are not files.... This is lack of knowledge. It is
also basically and fundamentally wrong to implement such features as
requested on streams: then they behave like files..... Which are not
streams..... <sigh and grumpy    >

To explain it for noobs or indifferent amateurs:
A stream is like a flow of water, you can not go back in that flow, it
just passed you, you could have caught the fish....
A stream is sequential and by its very nature unidirectional, not bi-
directional. Don't try to make it so. In that case it's a file....

I know this sounds strange, since some streams actually allow to set
position, but than it is simply a matter of the wrong name for stream.
It becomes a file....

Computer science basics...In any language...

----------------------------------------------------------------------
glorfin · 2017-08-02 09:10

Quote from: Thaddy on August 02, 2017, 11:01:09 amIt is also basically
and fundamentally wrong to implement such features as requested on
streams: then they behave like files.....
This feature is implemented long ago for any stream except Bufstream.
And I don't see wha we should impose artificial limitations just
because of name. We know anyway that TFileStream, for example, is just
a convenient superstructure on top of file. And, by the way, when you
are looking streaming video over internet, you nevertheless can rewind
it.

----------------------------------------------------------------------
Thaddy · 2017-08-02 09:15

A bufstream is actually a stream filter..... with sizeof(buffer) as
file.
A stream is infinite... a buffer is finite....
All computer science basics.
I know that you can rewind a "stream" but that is a stream filter, not
a stream. Amateurs or inadequately educated programmers confuse this.
So much so that you demand it? WRONG!@

----------------------------------------------------------------------
glorfin · 2017-08-02 09:22

Quote from: Thaddy on August 02, 2017, 11:15:26 amA bufstream is
actually a stream filter..... with sizeof(buffer) as file.
A stream is infinite... a buffer is finite....
All computer science basics

TBufStream is intended simply for buffered file input-output
operations. It owns TFileStream, reads first into own buffer and then
allows to read small chunks of data from this buffer. Size of Buffer
seldom is equal to the size of file.

However, I have a feeling that our discussion easily can become
excessively long and fruitless and by no means constructive. So, my
suggestion is that we both remain with own point of view.

----------------------------------------------------------------------
Blestan · 2017-08-02 09:26

streams by concept are not only unidirectional but also dont have size
... its not guaranteed in any circumstance that you can read an exact
amount of data ... even more in some environments streams dont even
wait for you to read the data ... you must plug in to the stream and
start reading if you discover some markers you can read some
reasonable data and analize it ... thats all...

seek(), eof bof etc functions are related to files .. even more to
databases...
note that in the early days the file system was a superset of the
database not like now when the  db is contained ( in most cases ) in a
file within the filesystem

----------------------------------------------------------------------
Blestan · 2017-08-02 09:29

@glorfin: if tbufstream own a filestream what is the point -> this is
a double buffering because the OS already buffers in most scenarios
all the open file handles

----------------------------------------------------------------------
marcov · 2017-08-02 09:57

Quote from: Thaddy on August 02, 2017, 11:01:09 amStreams are
streams... They are intended to stream in 1 (one) direction. They are
not files.... This is lack of knowledge. It is also basically and
fundamentally wrong to implement such features as requested on
streams: then they behave like files..... Which are not streams.....
<sigh and grumpy    >


Pure streams are not seekable (forward or backwards. In theory you
could emulate forward by reading dummy bytes) But I think you can
assume that streams that implement Seek are seekable.

----------------------------------------------------------------------
marcov · 2017-08-02 09:58

Quote from: Blestan on August 02, 2017, 11:29:29 am@glorfin: if
tbufstream own a filestream what is the point -> this is a double
buffering because the OS already buffers in most scenarios all the
open file handles

No it does not. Using a buffered stream instead of a non buffered one
can speed up many small reads/writes.   At least on Windows it does,
and the differences can be very pronounced.

----------------------------------------------------------------------
Thaddy · 2017-08-02 10:14

Quote from: marcov on August 02, 2017, 11:57:02 amBut I think you can
assume that streams that implement Seek are seekable.
Which means they implement a stream filter. The rest of your
explanation is as always correct. Files are seekable, streams are not.
It is just that somebody decided not to document it as such and
introduced a misnominer that is hard to get rid of.
In this case a TBufStream is a stream filter, not the actual stream.

----------------------------------------------------------------------
Blestan · 2017-08-02 10:38

@marcov: msnd :"  File buffering is usually handled by the system
behind the scenes and is considered part of file caching within the
Windows operating system unless otherwise specified. "

----------------------------------------------------------------------
wp · 2017-08-02 11:05

No, Blestan, the buffering by the OS is very ineffective (at least on
Windows). This is the output of the following test program on Win10:

QuoteCreating test file with 1000000 bytes... Done.
Reading byte by byte using a TMemorystream: 0.015 sec
Reading byte by byte using a TFileStream: 2.157 sec
Reading byte by byte using a TReadBufStream: 0.031 sec
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
Project1; {$mode objfpc}{$H+} uses  Classes, Bufstream, SysUtils; var 
stream: TStream;  bstream: TBufStream;  data: array of byte;  t:
TDateTime;  i: Integer; const  FILENAME = 'testdata.dat';  N =
1000000; begin  Write('Creating test file with ', N, ' bytes...'); 
stream := TFileStream.Create(FILENAME, fmCreate);  try   
setLength(data, N);  // 100 MB test file   
Stream.WriteBuffer(data[0], Length(data));  finally    stream.Free; 
end;  WriteLn(' Done.');   stream := TMemoryStream.Create;  try   
Write('Reading byte by byte using a TMemorystream: ');    t := Now;  
TMemorystream(stream).LoadFromFile(FILENAME);    for i := 0 to
stream.Size-1 do      stream.ReadByte;    t := Now - t;   
writeLn(FormatDateTime('s.zzz" sec"', t));  finally    stream.Free; 
end;   stream := TFileStream.Create(FILENAME, fmOpenRead);  try   
Write('Reading byte by byte using a TFileStream: ');    t := Now;   
for i := 0 to stream.Size-1 do      stream.ReadByte;    t := Now - t; 
  WriteLn(FormatDateTime('s.zzz" sec"', t));     stream.Position :=
0;    bstream := TReadBufStream.Create(stream, 1024);    try     
Write('Reading byte by byte using a TReadBufStream: ');      t :=
Now;      for i := 0 to stream.Size-1 do        bstream.ReadByte;    
t := Now - t;      WriteLn(FormatDateTime('s.zzz" sec"', t));   
finally      bstream.Free;    end;  finally    stream.Free;   
DeleteFile(FILENAME);  end;end.

----------------------------------------------------------------------
howardpc · 2017-08-02 11:16

Linux file buffering is noticeably better in this particular test.
Results for Mint 18.0

QuoteCreating test file with 1000000 bytes... Done.
Reading byte by byte using a TMemorystream: 0.017 sec
Reading byte by byte using a TFileStream: 0.177 sec
Reading byte by byte using a TReadBufStream: 0.013 sec

----------------------------------------------------------------------
Thaddy · 2017-08-02 11:21

@wp
@howardpc

That's the reason there is a TBufStream. Also note, old hands, that
buffers are only efficient when aligned and preferably either file
system default allocation size or a multitude of that. An arbitrary
buffer size can lead to performance loss instead of performance gain.
In the olden days I kept thinking 512,412,no 512, but I learned to
think 4096 already (omg) or worse...

----------------------------------------------------------------------
Blestan · 2017-08-02 11:49

the memorystrem example is not suitable for this case because it's
basicly only one big read from the disk and the just some move
operations
you sould aleast divide the timing s 2 one after the load from file
and one after the byte per byte operations.
btw a better approach in this case is to use memory mapped  io ... to
map the file and then just doing pointer operations

----------------------------------------------------------------------
wp · 2017-08-02 12:00

Memory stream is mentioned for no reason - forget it.

I don't understand the rest of the message (why don't you read it
again after sending it?). If you mean the time to open the filestream
- this is zero.

----------------------------------------------------------------------
glorfin · 2017-08-02 12:01

Quote from: Blestan on August 02, 2017, 12:38:00 pm@marcov: msnd :" 
File buffering is usually handled by the system behind the scenes and
is considered part of file caching within the Windows operating system
unless otherwise specified. "
Nevertheless, use of TReadBufStream for buffering of file read sped up
file reading 10 times in my case (from 34 seconds to 3.5 seconds). So,
it is definitely worthwhile.

----------------------------------------------------------------------
Blestan · 2017-08-02 12:15

@wp:  TMemorystream(stream).LoadFromFile(FILENAME);
does the actual reading from disk and costs the most of the time ...
readbyte calls are just move() operations

----------------------------------------------------------------------
wp · 2017-08-02 12:34

Yes, certainly. As I said, forget the MemoryStream data.

The main statement of the speed comparison test that I wanted to make
is that TBufStream is 1-2 orders of magnitude faster than "out-of-the-
box" TFileStream when reading small chunks of data (on Windows). Of
course, we don't know how slow TFileStream would be if the OS would
not buffer file access. But even then, TBufStream (or TMemoryStream if
the file is not too large) is worth considering.

----------------------------------------------------------------------
Blestan · 2017-08-02 12:37

and ofcourse if the file is <2GB memory mapping will be the fastest
access because is at the speed of the kernel io operations

----------------------------------------------------------------------
marcov · 2017-08-02 14:31

Quote from: Thaddy on August 02, 2017, 12:14:50 pmWhich means they
implement a stream filter. The rest of your explanation is as always
correct. Files are seekable, streams are not.
It is just that somebody decided not to document it as such and
introduced a misnominer that is hard to get rid of.
In this case a TBufStream is a stream filter, not the actual stream.

TFileStream is already seekable.

----------------------------------------------------------------------
ASerge · 2017-08-03 01:17

Quote from: Blestan on August 02, 2017, 02:37:51 pmand ofcourse if the
file is <2GB memory mapping will be the fastest access because is at
the speed of the kernel io operations
Yes. This test shows that BufStream is not so good, but the mem map is
the fastest (Window 7, x64)
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
Project1; {$mode objfpc}{$H+}{$APPTYPE CONSOLE} uses  Windows,
Classes, Bufstream, SysUtils; type  TFileMapRead =
class(TCustomMemoryStream)  private    FMapHandle: THandle;   
procedure CloseMapping;  public    constructor Create(const FileName:
string);    destructor Destroy; override;  end;  procedure
DoReadByByte(Stream: TStream);var  i: Integer;begin  for i := 0 to
Stream.Size - 1 do    Stream.ReadByte;end; procedure
WriteTimeDiff(From: TDateTime);begin  WriteLn(FormatDateTime('s.zzz"
sec"', Now - From));end; var  Stream: TStream;  BStream: TBufStream; 
Data: array of byte;  t: TDateTime; const  FILENAME = 'testdata.dat'; 
N = 1000000; constructor TFileMapRead.Create(const FileName:
string);var  FileSize: Integer;  HFile: THandle;  P: Pointer;begin 
inherited Create;  HFile := CreateFile(PChar(FileName), GENERIC_READ, 
  FILE_SHARE_READ, nil, OPEN_EXISTING,    FILE_ATTRIBUTE_NORMAL or
FILE_FLAG_SEQUENTIAL_SCAN, 0);  if HFile = INVALID_HANDLE_VALUE then 
 RaiseLastOSError;  FileSize := GetFileSize(HFile, nil);  if FileSize
> 0 then  begin    FMapHandle := CreateFileMapping(HFile, nil,
PAGE_READONLY, 0, 0, nil);    CloseHandle(HFile);    if FMapHandle = 0
then      RaiseLastOSError;    P := MapViewOfFile(FMapHandle,
FILE_MAP_READ, 0, 0, 0);    if P = nil then      RaiseLastOSError;   
SetPointer(P, FileSize);  end  else  begin    CloseHandle(HFile);   
SetPointer(nil, 0);  end;end;  procedure
TFileMapRead.CloseMapping;begin  if Memory <> nil then  begin   
UnmapViewOfFile(Memory);    SetPointer(nil, 0);  end;  if FMapHandle
<> 0 then  begin    CloseHandle(FMapHandle);    FMapHandle := 0; 
end;end; destructor TFileMapRead.Destroy;begin  CloseMapping; 
inherited;end; begin  Write('Creating test file with ', N, '
bytes...');  Stream := TFileStream.Create(FILENAME, fmCreate);  try  
SetLength(Data, N);  // 100 MB test file   
Stream.WriteBuffer(Data[0], Length(Data));  finally    Stream.Free; 
end;  WriteLn(' Done.');  Stream := TMemoryStream.Create;  try   
Write('Reading byte by byte using a TMemorystream: ');    t := Now;  
TMemorystream(Stream).LoadFromFile(FILENAME);   
DoReadByByte(Stream);    WriteTimeDiff(t);  finally    Stream.Free; 
end;  Stream := TFileStream.Create(FILENAME, fmOpenRead);  try   
Write('Reading byte by byte using a TFileStream: ');    t := Now;   
DoReadByByte(Stream);    WriteTimeDiff(t);    Stream.Position := 0;  
BStream := TReadBufStream.Create(Stream, 1024);    try     
Write('Reading byte by byte using a TReadBufStream: ');      t :=
Now;      DoReadByByte(Stream);      WriteTimeDiff(t);    finally    
BStream.Free;    end;  finally    Stream.Free;  end;  Stream :=
TFileMapRead.Create(FILENAME);  try    Write('Reading byte by byte
using a TFileMapRead: ');    t := Now;    DoReadByByte(Stream);   
WriteTimeDiff(t);  finally    Stream.Free;  end; 
DeleteFile(FILENAME);  Readln;end.QuoteCreating test file with 1000000
bytes... Done.
Reading byte by byte using a TMemorystream: 0.016 sec
Reading byte by byte using a TFileStream: 0.956 sec
Reading byte by byte using a TReadBufStream: 0.944 sec
Reading byte by byte using a TFileMapRead: 0.015 sec

----------------------------------------------------------------------
glorfin · 2017-08-03 02:26

No, it does not. Because when you say that you read with the help of
TBufStream, you actually read with TFilestream. Here is part of your
code:
Quote from: ASerge on August 03, 2017, 03:17:37 amCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i     BStream
:= TReadBufStream.Create(Stream, 1024);    try      Write('Reading
byte by byte using a TReadBufStream: ');      t := Now;     
DoReadByByte(Stream);  // <------------------- Here must be
DoReadByByte(BStream)      WriteTimeDiff(t);    finally     
BStream.Free;    end;

----------------------------------------------------------------------
SymbolicFrank · 2017-08-03 07:05

I agree with Blestran, just memory-map it. Especially because most
CPUs today are 64-bit.

Streams are so last century.

----------------------------------------------------------------------
Thaddy · 2017-08-03 08:04

Quote from: SymbolicFrank on August 03, 2017, 09:05:38 amI agree with
Blestran, just memory-map it. Especially because most CPUs today are
64-bit.

Streams are so last century.
That is simply not true. Streams are core to modern computing. It is
the thorough misunderstanding (stream vs stream filter) of what a
proper stream is that also leads to a thorough misunderstanding of its
application.
And this misunderstanding only seems the case amongst very average
Pascal programmers....  E.g.: A Pipe is also a stream and not
seekable. An UDP or TCP transfer is a stream and not seekable. A
stream of water from your tap is a stream and not seekable. A river is
a stream and not seekable. You can collect an amount of it and release
it back later at the correct spot if you employ a boey... That's a
filter, in this case a buffered stream... Get it?

Streams are 1) stateless, 2) unidirectional 3) sequential and 4) not
seekable. They can be made seekable by applying a filter: and that's a
different beast.

The problem is that somebody in the pascal community introduced the
misnomer. Which leads to expectations about a stream that can only be
achieved by a stream filter. Which can manifest itself, but not
necessarily -  as a buffer. A buffer is where you perform the seeks
on, not the stream. A buffer is stateful. Note memory mapping is a
buffer manifestation..... BASICS.

----------------------------------------------------------------------
SymbolicFrank · 2017-08-03 09:00

Stream buffers reside in main memory, so you would want to keep them
as small as possible if they have to fit into cache memory (L1 cache
for an i7 is 32kB).

Streams would be a good model if you skip the cache memory and only
use DMA (like a Cell SPE CPU). In that case, it works as intended.

In most other cases, you might want to use paging to move data around
between processes/threads, but that doesn't work on Windows (the OS
won't let you do it).

For sockets, I would simply limit the max size of a message and
read/write it in one go, bypassing the need for smaller buffers.

----------------------------------------------------------------------
Blestan · 2017-08-03 10:33

my opinion for memory mapped file is based on the fact that files are
not streams and as finite entity they can be easily read when mapped
to ram ...
otoh streams a very very different beasts and their goal, benefits
and  weaknesses   are a complete different story but they still have
and will have their place in modern programming ...

----------------------------------------------------------------------
Thaddy · 2017-08-03 10:58

Quote from: Blestan on August 03, 2017, 12:33:38 pmmy opinion for
memory mapped file is based on the fact that files are not streams and
as finite entity they can be easily read when mapped to ram ...
otoh streams a very very different beasts and their goal, benefits
and  weaknesses   are a complete different story but they still have
and will have their place in modern programming ...

Yes, you got it!

----------------------------------------------------------------------
Thaddy · 2017-08-03 11:01

Quote from: SymbolicFrank on August 03, 2017, 11:00:36 amStream
buffers reside in main memory
True, but you insist on not understanding streams. A stream buffer is
a stream filter... DON'T EVER CONFUSE THEM! (sorry about shouting, but
this is really really important!    )

----------------------------------------------------------------------
SymbolicFrank · 2017-08-03 11:09

Yes, yes, the difference between serial and block devices. I know all
that.

If I would design my own CPU, I would use the cache memory to
implement paged local storage. And do all communications through 4k
blocks or small system messages.


Anyway, even your phone is probably 64-bits and has at least a GB of
memory, so whatever.

----------------------------------------------------------------------
glorfin · 2017-08-03 12:57

Concerning intetesting experiment of ASerge, where an unfortunate
mistake was done (see my previous comment). I have repeated it without
this error.
Here is the result:

Creating test file with 1000000 bytes... Done.
Reading byte by byte using a TMemorystream: 0.016 sec
Reading byte by byte using a TFileStream: 1.420 sec
Reading byte by byte using a TReadBufStream: 0.016 sec
Reading byte by byte using a TFileMapRead: 0.000 sec

Which means that memory mapping was still better, but TBufStream sped
up file reading quite considreably, so its use is nevertheless
worthwhile.

----------------------------------------------------------------------
glorfin · 2017-08-03 13:06

Another note about the discussion as a whole.

I would agree, that TFileStream (as well as TBufSteam and on) may not
the best name for the objects we are discussing and word "stream" must
not be there. But it is already there and it is too late to speak
about it. These objects implement certain behaviour, and TStream was
seekabe already in Turbo Vision time. Now, function TBufStream.Seek
does exist and its implementation is clearly wrong, as it does not do
what is intended. Obviously, it should be somehow fixed. We can either
re-implement it in correct way (as I suggested) or remove it
whatsoever for the sake of terminological purity. But it would be
strange to leave an error unattended because the erroneously
implemented function should not theoretically exist.

----------------------------------------------------------------------
Blestan · 2017-08-03 13:19

Told you ))
Mapping is at the speed of kernel paging io and ofcourse properly
aligned

----------------------------------------------------------------------
glorfin · 2017-08-03 13:25

Quote from: Blestan on August 03, 2017, 03:19:37 pmTold you ))
Mapping is at the speed of kernel paging io and ofcourse properly
aligned
Well, but effect of TBufStream is also quite good, so, I don't think
we must completely forget and neglect it.
What slightly surprises me, the initial focus of my post - that there
is an error and it must be corrected, and that I suggest a working
code, somehow faded out it this lengthy and excessively emotional
discussion.

----------------------------------------------------------------------
Blestan · 2017-08-03 13:33

create a completely new class like TBufferedFile and post the new code
this will avoid any misunderstanding about streams

----------------------------------------------------------------------
taazz · 2017-08-03 13:37

add a enhancement request on the bug tracker to rename TxxxxStream to
TxxRiver, boats can go up and down on rivers so nobody should object
for them to have a working seek. and you will piss of some of the
smart asses

sorry to hard to resist.

----------------------------------------------------------------------
Blestan · 2017-08-03 13:44

humm  ok ... i acknowledge that i hard to resist

----------------------------------------------------------------------
glorfin · 2017-08-03 13:44

Quote from: Blestan on August 03, 2017, 03:33:02 pmcreate a completely
new class like TBufferedFile and post the new code  this will avoid
any misunderstanding about streams

A solution, too.

Although, as I have said, seekable "Streams" existed already in Turbo
Vision, which means that "noobs" and "average pascal programmers" are
actually those who designed TV, which is a bit funny.
I wrote this post because used TBufStream to speed up operations in my
own project, it gave 10-times effect: 3 seconds instead of 30, then
found that Seek did not work, fixed it and thought that it would be
better if I need not to correct it in every new FP installation. So,
why not to share the problem and solution with the community and
hopefully get it fixed in all future versions?

Effect you know. Personally for me it will be easier to write
TSeekableBufStream(TBufStream) with correct Seek function and forget
it all, so that no repeating fixes in new FP versions are needed,and I
have correct behaviour.

----------------------------------------------------------------------
Thaddy · 2017-08-03 14:01

Quote from: glorfin on August 03, 2017, 03:44:43 pmAlthough, as I have
said, seekable "Streams" existed already in Turbo Vision,
Which means the misnomer already exists since then. Doesn't make it
right... Btw those streams were very, very crude and tried to pack all
in one on a still half-baked language extension.
And yes, noobs - from my point of view - start about there.....  It
came about around the time when I was a teaching computer science
student at the faculty of sociology at the Erasmus University in
Rotterdam..(1982-1985)
I am still trying to teach. It is open to you if you are still want to
learn.... Seekable streams are a contradiction in terms.... As
explained.

----------------------------------------------------------------------
Blestan · 2017-08-03 14:06

@ thaddy: and i'm personally  very grateful for your comments and
suggestions

----------------------------------------------------------------------
taazz · 2017-08-03 14:11

Quote from: Thaddy on August 03, 2017, 04:01:33 pmQuote from: glorfin
on August 03, 2017, 03:44:43 pmAlthough, as I have said, seekable
"Streams" existed already in Turbo Vision,
Which means the misnomer already exists since then. Doesn't make it
right... Btw those streams were very, very crude and tried to pack all
in one on a still half-baked language extension.
And yes, noobs - from my point of view - start about there.....  It
came about around the time when I was a teaching computer science
student at the faculty of sociology at the Erasmus University in
Rotterdam..(1982-1985)
I am still trying to teach. It is open to you if you are still want to
learn.... Seekable streams are a contradiction in terms.... As
explained.
IF that bothers that much then redefine the term it is not like you
describe a natural law.

----------------------------------------------------------------------
glorfin · 2017-08-03 14:17

Quote from: Thaddy on August 03, 2017, 04:01:33 pm Seekable streams
are a contradiction in terms.... As explained.
I agree that TStream is not the best name for this class of objects.
However it exists and for compatibility we have to keep it.
Unfortunately, inappropriate terminology is not uncommon in science,
not only computer science. "Action potential" in physiology, for
example, which in fact is a process consisting of several ion
currents, resulting in fast changes of membrane potential. But
everybody knows what it is, and this process is called "action
potential" just by tradition.

Same is with TStream. Yes, this is not the best name for these
objects, but it is there. Objects themselves are nevertheless
convenient for use and we (well, at least I) are not going to neglect
them only because of inappropriate name.

And, after all. ("Stream" in English) <> ("TStream" in Pascal).

----------------------------------------------------------------------
SymbolicFrank · 2017-08-03 14:37

I read a discussion somewhere (I think on this forum) about the
efficiency of only paging in memory on-demand. So, if you start an
executable, there is no memory paged in at all. Every action,
including the loading of the executable itself will generate a whole
lot of page faults, which in turn will map a single page (4kB) each.

At the time this mechanism was designed (80386, or half-way in an
80286), memory was scarce, and the designers envisioned lots of
processes that all shared that.

Today we would (for example) use the larger page size of 2MB per page,
and give each process a few in advance (code, data, stack). That is
much more efficient. (The page tables are huge and all the management
takes a lot of CPU time.)

Streams are like that: they were a good interface, first for serial
devices and later for files, but we have much more resources nowadays.

----------------------------------------------------------------------
ASerge · 2017-08-03 18:29

Quote from: glorfin on August 03, 2017, 04:26:28 amNo, it does not.
Because when you say that you read with the help of TBufStream, you
actually read with TFilestream. Here is part of your code:
Thank you. My mistake . After correcting the test shows a good
performance for BufStream
QuoteCreating test file with 1000000 bytes... Done.
Reading byte by byte using a TMemorystream: 0.017 sec
Reading byte by byte using a TFileStream: 0.944 sec
Reading byte by byte using a TReadBufStream: 0.018 sec
Reading byte by byte using a TFileMapRead: 0.013 sec

----------------------------------------------------------------------
SymbolicFrank · 2017-08-03 20:56

Btw, ironically enough, most current CPUs spend the vast amount of
their time either sleeping or waiting on the very slow main memory.
Your average i7 spends on average 90,000 clock cycles waiting for a
single byte to arrive from main memory. In that time, it could execute
up to 720,000 instructions...

Then again, it is very unlikely it could write the results of those
instructions back in any meaningful way, and accessing more than one
byte sequentially is much faster.

On the other hand, your working set for all processes/threads
combined, spread out over all processors in the system is just 32kB,
if you use shared memory...

And the most important thing you want in cache memory is your page
tables! Which have become huge!


A double page fault requires reloading that part of the page table
from disk, which is so excessively slow that it will take many
millions of cycles.

Then again, cache memory already fills more than 75% of the die of a
CPU, and suffers seriously from diminishing returns.


In short, the best ways to speed up processing are:

1. Faster and/or wider memory access.
2. Using an SSD.

But, structurally, you want to get rid of the slow memory dictating
speed, and use that die area occupied by cache memory in a more
efficient way.

The best way to do that, is to regard hard disks as really slow
memory, SSD's as very slow memory, RAM as slow memory and use the
cache memory as main memory.

To explain: instead of just a few very fat and starved cores, you want
many lean and mean ones. Get rid of registers as we know them, and
give them multiple kilobytes of local storage, that is at least as
fast as L2 cache memory, or even L1.

But, you want to be able to swap processes/threads, and send all the
other processes/threads extensive messages, both which require you to
be able to swap that local storage, fast.

Or, in other words: treat main memory and anything slower as a slow
block device (map global page 087243658 to local page 24), and use
paging to give all CPU cores a nice block of really fast local storage
(previously cache memory).

And for synchronizing stuff, you create a build-in messaging on the
system and chip level. Just a few words. Alerts.


But, in the mean time, we have to make do and use main memory as the
best we have.

----------------------------------------------------------------------
glorfin · 2017-08-03 21:03

Quote from: ASerge on August 03, 2017, 08:29:18 pm After correcting
the test shows a good performance for BufStream
QuoteYes, it shows slightly better performance of mapping, but
difference is not that dramatic.
Anyway, thank you for your experiment!