Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
MortenB · 2021-03-02 03:53

I would like to subscribe to certain feeds of data, and then display
this data on my computer,
What I really would like now is to make an app that listens to the
Binance Price-ticker, and simply displays the numbers, or everything
which is received.

Does anybody have information regarding which WebSocket addon I would
need, and an example of how to set this up?

I need to start a subscription.
Then I need to be able to read the returned stream.
And then close the subscription.
This just to get started. I learn best from relevant examples.

Thank you in advance.

----------------------------------------------------------------------
Jurassic Pork · 2021-03-02 05:57

hello,
you can try lazWebsockets !

Friendly, J.P

----------------------------------------------------------------------
MortenB · 2021-03-02 07:36

I have installed this Lazarus WebSocket-solution.
The problem is a total lack of guides as in how to use it against
various providers of data one can subscribe to.
I find no real examples of anything for this, except the provided
example of a chat server and client.

----------------------------------------------------------------------
Warfley · 2021-03-02 21:33

The library does only implement the base websocket protocol. Your
subscription system must use some form of sub protocol on top of
websockets, this is up to you to implement. The library just gives you
the networking capabilities to establish streams and send and recieve
messages.

The service you are subscribing to needs to have some form of
documentation. If you find that, maybe I can help you with specific
questions.

----------------------------------------------------------------------
PierceNg · 2021-03-03 00:21

Quote from: Warfley on March 02, 2021, 10:33:16 pmThe library does
only implement the base websocket protocol. Your subscription system
must use some form of sub protocol on top of websockets, this is up to
you to implement. The library just gives you the networking
capabilities to establish streams and send and recieve messages.

The service you are subscribing to needs to have some form of
documentation. If you find that, maybe I can help you with specific
questions.

OP mentioned Binance, so possibly this one:
https://github.com/binance/binance-spot-api-docs/blob/master/web-
socket-streams.md

----------------------------------------------------------------------
MortenB · 2021-03-03 07:36

Thank you for your time and effort in this matter.
And yes.
I did mention Binance, and of course there is a lot of things I read
from the Binance API.
Everything there seems to be in Joson-format.
And... making the Joson-queries and reading the Joson-answers is a
piece of cake. I have no problems there, and I believe the API is well
enough documented for me to work with that.
-
However.
What I have never done before is established a subscription.
Or sent any messages at all via the Internet.
What I really would like is:
1. An example of how to send messages/streams/strings to the service
provider
2. An example of how to know if I received a reply, errormessage or
anything that I can interpret.
3. An example of how to read the received stream.
-
In my simple mind, I consider the workflow to be like:
1. Subscribe to DataStream
2. Check for errormessage.
3. If no errormessage, then check if new data is available on/in the
stream 3–4 times every second.
4. If data is available, read and display information.
and then just cycle 3 and 4.
-
-
It doesn't have to be using the Lazarus WebSockets.
I also see that Indy has all the functionality for WebSockets, among
other things, and from what I read, this Indy is now very very
stable. Indy is maybe the better choice?

----------------------------------------------------------------------
Warfley · 2021-03-03 13:20

Well, the chatClient example should contain all the information you
need, but let me break it down for you.

First you need to open up the websocket stream, represented by the
TWebsocketCommunincator class:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i client :=
TWebsocketClient.Create(address, port, path); // e.g. for
wss://example.com:8080/foo?bar=foobar address would be example.com
port would be 8080 (default 80) and path would be /foo?bar=foobar
(default /)  try    wsstream := client.Connect(TSocketHandler.Create);
// Use an SSL socket handler for SSL, there are examples in this forum
how to do this    try      ...    finally      wsstream.Free;    end; 
finally    client.Free;  end;Next step is to send a json string:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
wsstream.WriteStringMessage(jsonString);For recieving messages just
call:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
wsstream.RecieveMessage;This function will block until a message was
read. Messages read by it will be stored in the thread safe
wsstream.Messages datastructure. To read all recieved messages call
"GetUnprocessedMessages".

So if you expect exactly one response message and you want to wait for
it blocking:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
RecieveResponse(wsstream: TWebsocketCommunicator): String;var 
msgList: TWebsocketMessageOwnerList;  msg: TWebsocketMessage;begin 
Result := nil;  msgList := TWebsocketMessageOwnerList.Create;  try   
while wsstream.Open and not Assigned(Result) do    begin     
wsstream.Recievemessage;      msgList.Clear;      if
wsstream.GetUnprocessedMessages(msgList) > 0 then        for msg in
msgList do          if msg is TWebsocketStringMessage then           
Exit(TWebsocketStringMessage(msg).Data);    end;  finally   
msgList.Free;  end;end;This function will wait for and recieve the
very first response that is a string and discards all other messages
(binary messages or pong messages)

If you want to recieve messages asynchonously, i.e. you don't want to
explicitly wait for messages, you can do so by setting the
OnRecieveMessage event handler and execute Recievemessage in a
seperate thread:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
TRecieverThread = class(TThread)  private    FCommunicator:
TWebsocketCommunincator;  protected    procedure Execute; override; 
public    constructor Create(ACommunicator: TWebsocketCommunincator); 
end; ... procedure TRecieverThread.Execute;begin  while not Terminated
and FCommunicator.Open do  begin    FCommunicator.RecieveMessage;   
Sleep(100);  end;end; constructor
TRecieverThread.Create(ACommunicator: TWebsocketCommunincator);begin 
FCommunicator := ACommunicator;  inherited Create(True); 
FreeOnTerminate := True;end; ...  wsstream.OnRecieveMessage :=
@MessageHandler;  TRecieverThread.Create(wsstream).Start;Now the
thread will automatically check for messages in the background and
when one is recieved, MessageHandler will be called. BUT:
MessageHandler will be called from within the context of the Reciever
thread. So if you want to, for example, display the message in your
GUI, you need to use synchronize or TThread.Queue:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i TMyClass =
class  LatestMessage: String;  procedure UpdateUI;  procedure
MessageHandler(Sender: TObject); ... procedure
TMyClass.UpdateUI;begin  Label1.Caption :=
LatestMessage;end; procedure TMyClass.MessageHandler(Sender:
TObject);beginvar  MsgList: TWebsocketMessageOwnerList;  m:
TWebsocketMessage;begin  LatestMessage := nil;  MsgList :=
TWebsocketMessageOwnerList.Create(True);  try   
TWebsocketCommunicator(Sender).GetUnprocessedMessages(MsgList);    for
m in MsgList do      if m is TWebsocketStringMessage then       
LatestMessage := TWebsocketStringMessage(m).Data;  finally   
MsgList.Free;  end;  if Assigned(LatestMessage) then   
Synchronize(@UpdateUI);end;TMyClass can for example be your TForm
derivate you are working in, if you are working in a GUI application.

----------------------------------------------------------------------
MortenB · 2021-03-03 14:31

Quote from: Warfley on March 03, 2021, 02:20:45 pmWell, the chatClient
example should contain all the information you need, but let me break
it down for you.

Thank you so much for the "breakdown" 
I think I can handle this pretty well.

It is too late for me tonight, but if I have time, I will try to do
something tomorrow.
Ideally, what I would like to do is to make my form wait for input
from mouse and keyboard at all times.
a timer will be set to check for messages maybe 10 times per second
(just an example.)
Then if the timed "interrupt" find received message(s), they will be
dealt with before letting go.

Your "breakdown" of this is great. If I encounter problems, I will try
to ask for more details.
Thank you!

----------------------------------------------------------------------
Warfley · 2021-03-03 15:10

After writing this post I noticed that this is a little bit
overcomplicated, so I just added some new functions to the library
that makes most of what I wrote previously invalid.

You now don't need to call RecieveMessage anymore. If you are waiting
for a message and want to block until you have it, you can now use
WaitForMessage or if you are awaiting a special kind of message
WaitForStringMessage, WaitForBinaryMessage and WaitForPongMessage. In
your case, awaiting the response will therefore look something like
this:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i msg :=
wsstream.WaitForStringMessage;try  if not Assigned(msg) then // stream
closed (disconnect, server closed stream, etc.) before message was
recieved    // Error handling  jsonString := msg.Data;finally 
msg.Free;end;This is very useful if you are expecting a response, e.g.
when you do your subscription and need to check if it was successful.

You also now don't need to create your own TThread instance for
recieving messages, simply call wsstream.StartRecieveMessageThread and
a new thread will be started recieving messages (see the Readme for
further information about threading).

Most importantly, WaitForXXXMessage works fine in both situations, if
you have a reciever thread and if you don't.

QuoteThen if the timed "interrupt" find received message(s), they will
be dealt with before letting go.This is also quite simple, first start
the reciever thread, then do the following in your timer:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TForm1.Timer1Timer(Sender: TObject);var  Messages:
TWebsocketMessageOwnerList;  Message: TWebsocketMessage;begin 
Messages := TWebsocketMessageOwnerList.Create;  try   
wsstream.GetUnprocessedMessages(Messages);    for Message in Messages
do      // Update your UI with the contents of Message  finally   
Messages.Free;  end;end;
This is also completely thread safe, so you don't need to worry about
threading issues (as would be the case when using the event
OnRecieveMessage)

----------------------------------------------------------------------
bytebites · 2021-03-04 11:26

Recieve should be written as receive.

----------------------------------------------------------------------
MortenB · 2021-03-05 14:31

Quote from: Warfley on March 03, 2021, 02:20:45 pmFirst you need to
open up the websocket stream, represented by the
TWebsocketCommunincator class:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i client :=
TWebsocketClient.Create(address, port, path); // e.g. for
wss://example.com:8080/foo?bar=foobar address would be example.com
port would be 8080 (default 80) and path would be /foo?bar=foobar
(default /)  try    wsstream := client.Connect(TSocketHandler.Create);
// Use an SSL socket handler for SSL, there are examples in this forum
how to do this    try      ...    finally      wsstream.Free;    end; 
finally    client.Free;  end;I have to admit. SSL Socket handler is
also new to me. I have no clue.
Basically what I end up with when I try to compile the line: "wsstream
:= client.Connect(..."
is an errormessage: Fatal: Syntax error, "." expected but ":=" found.

I started with a button to start subscription
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i client :=
TWebsocketClient.Create('wss://stream.binance.com', 9443, '/');and
should send a JSON-string to request subscription data when connection
has been confirmed.
{
    "type": "subscribe",
    "exchange": "binance",
    "pair": "eth-btc",
    "channel": "trade"
}

My simple idea here was to start a timer that constantly checks if a
message is received, and then displays whatever is received in an
edit-box or something.

Then a button to stop subscription

----------------------------------------------------------------------
Warfley · 2021-03-05 17:05

Quote from: MortenB on March 05, 2021, 03:31:34 pmI have to admit. SSL
Socket handler is also new to me. I have no clue.
I think you can simply use the class method
"TSSLSocketHandler.GetDefaultHandler" from the unit sslsockets to use
ssl
Quote from: MortenB on March 05, 2021, 03:31:34 pmBasically what I end
up with when I try to compile the line: "wsstream :=
client.Connect(..."
is an errormessage: Fatal: Syntax error, "." expected but ":=" found.
Sorry, I am stupid, wsstream is the name of one of my units, so you
need to name the variable differently.

Quote from: MortenB on March 05, 2021, 03:31:34 pmI started with a
button to start subscription
and should send a JSON-string to request subscription data when
connection has been confirmed.
{
    "type": "subscribe",
    "exchange": "binance",
    "pair": "eth-btc",
    "channel": "trade"
}

My simple idea here was to start a timer that constantly checks if a
message is received, and then displays whatever is received in an
edit-box or something.

Then a button to stop subscription

I recommend the following, put the client and stream variable into you
Form (e.g. as private variables). In FormCreate create both variables
and set up events:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Self.Client
:= TWebsocketClient.Create('stream.binance.com', 9443,
'/');Self.Stream := Self.Client.Connect(TSSLSocketHandler.GetDefaultHa
ndler);Self.Stream.OnClose := @HandleClose; // Handle close is a
procedure HandleClose(sender: TObject); in your
classSelf.Stream.StartReceiveMessageThread; // handle all incoming
messagesAnd free them in the FormDestroy event:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Self.Stream.Free;Self.Client.Free;
Then in your subscribe button you start the subscription by sending
your json request and awaiting the correct answer:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Self.Stream.WriteStringMessage('JSON subscribe request');response :=
Self.Stream.WaitForStringMessage;try  // Check if response.Data
contains the expected responsefinally  response.Free;end;In your
unsubscribe button you do the same with the unsubscribe request

In your timer you then put the loop to check for received messages
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
procedure TForm1.Timer1Timer(Sender: TObject);    var      Messages:
TWebsocketMessageOwnerList;      Message: TWebsocketMessage;    begin 
    Messages := TWebsocketMessageOwnerList.Create;      try       
wsstream.GetUnprocessedMessages(Messages);        for Message in
Messages do          // Update your UI with the contents of Message  
  finally        Messages.Free;      end;    end;
Also note, that I had updated the code today, so I highly recommend
you downloading the newest version. Best to load it via git so you can
easiely check for updates using git pull

----------------------------------------------------------------------
MortenB · 2021-03-07 14:05

I have now been back and forth a bit with the code, but there are
issues I cannot easily solve here.
First thing first, a tip: "TWebSocketCommunincator", should this not
be "TWebSocketCommunicator"? (Just a typo, maybe intended, I don't
know.)
-
Then my big problem.
First my code:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses 
Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,
ExtCtrls,  wsutils,  wsmessages,  wsstream,  ssockets, 
WebsocketsClient; type   { TForm1 }   TForm1 = class(TForm)   
btnSubscribe: TButton;    btnUnsubscribe: TButton;    edtPort: TEdit; 
  edtSubscribeAddress: TEdit;    mmoResult: TMemo;   
mmoSubscriptionInfo: TMemo;    MessageTimer: TTimer;    procedure
btnSubscribeClick(Sender: TObject);    procedure FormCreate(Sender:
TObject);    procedure FormDestroy(Sender: TObject);    procedure
MessageTimerTimer(Sender: TObject);  private    MyClient   :
TWebsocketClient;    MyStream   : TWebSocketCommunincator;//TStream; 
 procedure HandleClose(sender: TObject);  public  end; var  Form1:
TForm1;  MyStreamVar : TLockedSocketStream;implementation {$R *.lfm} {
TForm1 } procedure TForm1.HandleClose(Sender: TObject);  begin 
end; procedure TForm1.FormCreate(Sender: TObject);  begin   
MyStream.Create(MyStreamVar,False,False);    MyClient :=
TWebsocketClient.Create('stream.binance.com', 9443, '/');    MyStream
:= MyClient.Connect(TSocketHandler.Create);// GetDefaultHandler   
MyStream.OnClose := @HandleClose; // Handle close is a procedure
HandleClose(sender: TObject); in your class   
MyStream.StartReceiveMessageThread; // handle all incoming messages 
end; procedure TForm1.btnSubscribeClick(Sender: TObject);  VAR   
MyResponse : TWebSocketStringMessage;    MyTextData : UTF8String; 
begin    {    MyTextData:='';    MyResponse.Create(MyTextData);   
MyStream.WriteStringMessage(mmoSubscriptionInfo.Lines.Text);// 'JSON
subscribe request'    MyResponse := MyStream.WaitForStringMessage;   
try      // Check if response.Data contains the expected response   
finally      MyResponse.Free;    end;    }  end; procedure
TForm1.FormDestroy(Sender: TObject);  begin    MyStream.Free;   
MyClient.Free;  end;  procedure TForm1.MessageTimerTimer(Sender:
TObject);  var    Messages: TWebsocketMessageOwnerList;    Message:
TWebsocketMessage;  begin    {    Messages :=
TWebsocketMessageOwnerList.Create;    try     
MyStream.GetUnprocessedMessages(Messages);      for Message in
Messages do        // Update your UI with the contents of Message    
mmoResult.Append(Message.ToString);    finally      Messages.Free;   
end;    }  end;end.So far, I get no error on compile.
When running, I get SIGSEGV on: MyStream.OnClose := @HandleClose;
When I // don't do this line, I get SIGSEGV on the next line,
MyStream.StartReceiveMessageThread;, which then point to line 645 in
the wsstream-unit:   if not Assigned(FReceiveMessageThread) then...

There just seems to be an endless list of things popping up here.
Maybe because I don't have the right units, maybe because I don't know
how to initialize the variables properly.
I realize that this wasn't a walk in the park  But I don't want to
give up 

I don't even know if I need to install something for SSL-
functionality.
Your example gives: TSSLSocketHandler
But I cannot seem to find this here.

----------------------------------------------------------------------
MortenB · 2021-03-08 05:43

I finally found out about the TSSLSocketHandler.GetDefaultHandler.
Hoorayyy. One step further!
Had to add sslsockets and OpenSSLsockets to the Uses 
But still. The problem persist when I start the handling of incoming
messages: X.StartReceiveMessageThread.
It goes to line 645 in the wsstream-unit and ends with a SIGSEGV
The exact error: Project xxx raised exception class 'External:
SIGSEGV'. In file wsstream.pas at line 645: if not
Assigned(FReceiveMessageThread) then
-
I hope you can point me in the right direction here.
Thanks in advance.

----------------------------------------------------------------------
Warfley · 2021-03-08 15:20

This problem is because the server declined the handshake, in this
case Client.Connect will return nil, and when trying to work on that,
you get an access violation.

To be precise, the server returns http code "400: Bad request".
Probably something I should throw an exception over, is on my todo
list. But for now, it seems like you are doing a faulty request.
Probably because you try to access the path '/', while the github doc
states:
QuoteRaw streams are accessed at /ws/<streamName>
Combined streams are accessed at
/stream?streams=<streamName1>/<streamName2>/<streamName3>so your path
is probably something like '/ws/streamname' or '/stream?streams=...'

PS: the line
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
MyStream.Create(MyStreamVar,False,False);Is completely out of place
and should be removed

----------------------------------------------------------------------
MortenB · 2021-03-09 06:45

There are a few issues, yes.
I have tried and tried various approaches to achieve a connection and
not get the SIGSEGV-error.
In the end, I installed a Chrome Addon with Direct Web Sockets
Functionality:
  https://chrome.google.com/webstore/detail/smart-websocket-
client/omalebghpgejjiaoknljcfmglgbpocdp/related
This to test the address that I try to use via Lazarus.
-
First things first.
The Chrome Addon, with the address:
  wss://stream.binance.com:9443/ws/btcusdt@ticker
connects very well and instantly starts receiving  the tickers in
JSON-format
{
  "e": "24hrTicker",
  "E": 1615268894603,
  "s": "BTCUSDT",
  "p": "3394.80000000",
  "P": "6.697",
  "w": "51548.22027742",
  "x": "50688.12000000",       ... and so on ...
–
However.
When I try to do the same with your WebSocket-implementation, I cannot
get it to work.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
MyClient := TWebsocketClient.Create('wss://stream.binance.com',9443,'/
ws/btcusdt@ticker');  does not produce an error, but the next step:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
MyCommunicator :=
MyClient.Connect(TSSLSocketHandler.GetDefaultHandler);gives a Debugger
Exception Notification: Project xxx raised exception class
'ESocketError' with message: Host name resolution for
"wss://stream.binance.com" failed.
It basically failes at line 74: ASocket.Connect; in the
WebsocketsClient-file
-
I have performed a nslookup from a command window:
nslookup wss://stream.binance.com
and the answer is 52.198.117.1 and 54.65.212.75
-This is the exact same address as ASocket.Connect cannot resolve.
–
Unfortunately, from there and on, I am in deep waters,
–
More things I have tried.
Deactivated my AntiVirus.
Deactivated Windows Firewall.
Neither of which changes the outcome.

----------------------------------------------------------------------
Warfley · 2021-03-09 09:01

Have you tried to not write "wss://" in front of the hostname?
i.e. TWebsocketClient.Create('stream.binance.com',9443,'/ws/btcusdt@ti
cker')

The reason for this is simple, the wss:// is not part of the host, but
is part of the URI. But, this library doe not work with URIs

----------------------------------------------------------------------
MortenB · 2021-03-09 09:16

I have tried just about everything, also leaving out the "wss://"
When leaving out "wss://" it ends up with external SIGSEGV in
wsstream.pas at line 645 as earlier mentioned.
-
Sad. I need to work with URIs.
Do you have any "immediate" plans to include this in your library?

----------------------------------------------------------------------
Warfley · 2021-03-09 10:07

I've found the problem. The biance server used a lowercase value for
the connection header, which is not explicitly allowed by the HTTP
standard (but is by the websockets standard), and therefore was not
implemented by me that way.
I updated the code to allow for this. Now the following example works
completely fine:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses 
Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,
ExtCtrls,  opensslsockets, sslsockets, WebsocketsClient, wsstream,
wsmessages; type   { TForm1 }   TForm1 = class(TForm)    Memo1:
TMemo;    Timer1: TTimer;    procedure FormCreate(Sender: TObject);  
procedure FormDestroy(Sender: TObject);    procedure
Timer1Timer(Sender: TObject);  private    FClient: TWebsocketClient; 
 FStream: TWebsocketCommunincator;  public   end; var  Form1:
TForm1; implementation {$R *.lfm} { TForm1 } procedure
TForm1.FormCreate(Sender: TObject);begin  FClient := TWebsocketClient.
Create('stream.binance.com',9443,'/ws/btcusdt@ticker');  FStream :=
FClient.Connect(TSSLSocketHandler.GetDefaultHandler); 
FStream.StartReceiveMessageThread;end; procedure
TForm1.FormDestroy(Sender: TObject);begin  FStream.Free; 
FClient.Free;end; procedure TForm1.Timer1Timer(Sender: TObject);var 
MsgList: TWebsocketMessageOwnerList;  Message:
TWebsocketMessage;begin  MsgList :=
TWebsocketMessageOwnerList.Create;  try   
FStream.GetUnprocessedMessages(MsgList);    for Message in MsgList do 
  begin      if not (Message is TWebsocketStringMessage) then       
Continue;     
Memo1.Lines.Add(TWebsocketStringMessage(Message).Data);    end; 
finally    MsgList.Free;  end;end;
PS: HTTP does not know about URIs so from a technical standpoint
including wss:// does not make any sense. This is just so general
purpose applications like JS can distinquish between websocket and
HTTP URIs, but if you start instanciating a websocket client, it
should already be pretty obvious that this is not HTTP. Also the
second s in wss means using SSL (i.e. a TSSLSocketHandler), so if you
have a ws:// uri you don't need ssl, if you have wss:// you need ssl.
But this is all handled seperately in the code, so there is no need
for URIs

----------------------------------------------------------------------
MortenB · 2021-03-09 16:41

Quote from: Warfley on March 09, 2021, 11:07:40 amI updated the code
to allow for this. Now the following example works completely fine:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
usesprocedure TForm1.FormCreate(Sender: TObject);begin  FClient := TWe
bsocketClient.Create('stream.binance.com',9443,'/ws/btcusdt@ticker'); 
FStream := FClient.Connect(TSSLSocketHandler.GetDefaultHandler); 
FStream.StartReceiveMessageThread;end; 
Thank you for your quick reply and update... but alas. No success on
my end.
There is still a problem when the third line: 
 FStream.StartReceiveMessageThread;   is executed.
Then I get the errormessage: Project xxx raised exception class
'EWebsocketReadError' with message: error reading from stream... in
file wsstream.pas at line 466: Stream.LastError);

However.
If I do a step by step approach, and step into the lines, then the
connection is Ok. The stream gets a few seconds to "fill up", and when
the timer kicks in, the Memo is filled with Binance Ticker-data.
And then... when the stream is empty, it crashes: At least that is
what I had here a couple of times, but I can't even reproduce this
now.

–
And again. I had an Idea.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TWebsocketCommunincator.ReceiveMessage: TWebsocketMessage;   procedure
ReadData(var buffer; const len: int64);  var    ToRead: longint;   
Read: longint;    LeftToRead: int64;    TotalRead: int64;    oldTO:
integer;    Stream: TSocketStream;  const    IOTimeoutError = {$IFDEF
UNIX}11{$ELSE}10060{$EndIf};    WaitingTime = 10;  begin   
Sleep(1000);    TotalRead := 0;    repeat      // how much we are
trying to read at a time      LeftToRead := len - TotalRead;      if
LeftToRead > ToRead.MaxValue then        ToRead := ToRead.MaxValue   
 else        ToRead := LeftToRead;      // Reading       Stream :=
FStream.LockRead;      try        if not Assigned(Stream) then       
begin          raise EWebsocketReadError.Create('Socket already
closed', 0);        end;        oldTO := Stream.IOTimeout;       
Stream.IOTimeout := 1;        try          Read :=
Stream.Read(PByte(@buffer)[TotalRead], ToRead);          if Read < 0
then          begin            // on Error            if
Stream.LastError <> IOTimeoutError then              raise
EWebsocketReadError.Create('error reading from stream',              
Stream.LastError);          end          else          begin         
 // Increase the amount to read            TotalRead += Read;        
end;        finally          Stream.IOTimeout := oldTO;        end;  
  finally        FStream.UnlockRead;      end;      if (TotalRead <
len) and (Read <> ToRead) then // not finished, wait for some data   
   Sleep(WaitingTime);    until TotalRead >= len;  end;As you notice,
I added Sleep(1000) in your ReceiveMessage-function.
As a less than elegant solution, this solves the problem, and I can
now listen to the stream. No more errors...
I really would like more control over the error-situation.
How about replacing the Raise EWebsocketReadError with simply a
message? like: "!No data yet!", and that way, when I look for the
JSON-strings, I can just avoid the "!No data yet!"-strings, or even
count them, and if too many occur on a row, then I can close and
restart the connection automatically.
I also see that you have a Stream.IOTimeout=1 in there. How about
changing this so that this is a property that can be changed, but that
it defaults to 1.
Not sure what 1 is, maybe 1 millisecond.

----------------------------------------------------------------------
Warfley · 2021-03-09 21:22

The thing with IOTimeout is that this is intendet to be cought:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i           
if Stream.LastError <> IOTimeoutError then              raise
EWebsocketReadError.Create('error reading from stream',              
Stream.LastError);The idea is, as this function acquires some locks,
to only read what data is there and if there was a timeout to simply
wait a little bit and try again, to not block all the locks until the
whole data has been read. So if no data is there yet, it should behave
exactly as you want it to, it simply waits a bit and tries again. 1 is
the minimum timeout (1 ms) so it basically just reads  what the OS has
buffered and does not wait for anything

Could you check out what the actual value of Stream.LastError is and
what OS you are using (these error codes are OS specific, e.g. 10060
on windows is the timeout error code).

----------------------------------------------------------------------
MortenB · 2021-03-10 02:32

Not being able to directly see the error-code from the
Stream.LastError, I changed the code slightly:
I added an extra variable.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var 
ErrorNumber : LongInt;
and then:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i           
if Stream.LastError <> IOTimeoutError then BEGIN             
ErrorNumber := Stream.LastError;              raise
EWebsocketReadError.Create('error reading from
stream',Stream.LastError);            END; After the break or raised
error:
When I then mouse-over IOTimeoutError, I get the value 10060, just to
show that the variable values are showing when debugging.
The same mouse-over on ErrorNumber, gives 0 or Zero.

My Os is Windows 10

----------------------------------------------------------------------
Jurassic Pork · 2021-03-10 06:27

hello,
with this code in the function TWebsocketCommunincator.ReceiveMessage
of the file wsstream.pas, it seems to be OK :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i        try 
        Read := Stream.Read(PByte(@buffer)[TotalRead], ToRead);      
  if Read < 0 then          begin            // on Error           
 if (Stream.LastError <> IOTimeoutError) and               
(Stream.LastError <> 0) then               raise
EWebsocketReadError.Create('error reading from stream',              
     Stream.LastError);          end          else          begin    
      // Increase the amount to read            TotalRead += Read;   
     end;        finally          Stream.IOTimeout := oldTO;       
end;
Windows 10 Lazarus 2.0.12 fpc 3.2.0

friendly, J.P

----------------------------------------------------------------------
Warfley · 2021-03-10 15:23

Well, this is really weird, because error number 0 means there was no
error, maybe some weird quirk of the windows api, but alas, I added
the (error <> 0) condition as it seems to not break anything

----------------------------------------------------------------------
MortenB · 2021-03-10 17:02

Great. I was going to add that myself here, then I saw The friendly
"Pork" here suggest it too (Thank you Pork, much appreciated), but I
wanted to wait for your judgement on this too

I have not had time to check yet how it works here, but if it works,
then I believe I can celebrate a little.

Thank you very much.

My next step is to make REST-calls using GET and/or PUT, but I suppose
that should be in a different thread altogether.
Anyway.
Thank you very much again!

----------------------------------------------------------------------
MortenB · 2021-03-13 06:28

Everything here works very well now.
All tested at my side now and I see no problems with the reception of
streams anymore.

Thank you again for your great help

----------------------------------------------------------------------
MortenB · 2021-04-13 15:29

After testing a lot with the WebSocket-solution, I was trying to ramp
up the size of receiving text.
And this... not functional in a meaningful manner.
I cannot say exactly where it goes wrong (sizewise), but text in the
order of half a megabyte seems to be not possible.
The file/unit "wsstream" constantly reports RunError 201 at line 563:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i         if
Header.PayloadLen < 126 then          len := Header.PayloadLen       
else if Header.PayloadLen = 126 then        begin         
ReadData(len16, SizeOf(len16));          len := NToHs(len16);       
end        else        begin          ReadData(len64, SizeOf(len64)); 
        len := ntohll(len64);   // Recurring errors here when the size
is "substantial".          // Always RunError 201        end;       
if Header.Mask then        begin          ReadData(MaskRec.Key,
SizeOf(MaskRec.Key));        end When I break and check the value of
"len", some times it is "0" = Zero
Other times it is like "8442272860644245284" (Actual example copied
out from the "Watches"-window.)
Sometimes my program runs for 1 second after I enable the websockets,
other times it just crashes right away.
When it runs for 1 second, I do receive ONE String.
During debug I have found the size to be around 303642–306021 bytes,
but most often it just crashes right away.

The line Code: Pascal  [Select][+][-]window.onload = function(){var x1
= document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
ReadData(len64, SizeOf(len64));During debug I can find that the value
of len64 is -3392894971223736320

I took my chances, and thought that Maybe, the len64 was declared
wrongly in the VAR-section.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var 
Header: TWebsocketFrameHeader;//  len64: int64;  len64: QWord; //
Mortens attempt 13.4.2021 to fix error below at line 563  len16:
word;  len: int64;  MaskRec: TMaskRec;  buffer: TBytes;
I changed the int64 to QWord, and Voila.
Now I can receive the huge chunks of texts, seemingly without errors.
I have NO IDEA if my change might cause different problems, but this
solved my immediate problem.
Maybe Warfley has a comment on my fix?
Is it an Ok fix? or should the problem be fixed elsewhere?
Thanks in advance

----------------------------------------------------------------------
Warfley · 2021-04-14 08:42

The reason the len64 is negative is because it is in network byte
order, not host byte order. On an x86(_64) system this means the bytes
need to be swapped, this is what ntohll (network to host long long) is
for.

But the runtime error is really weird, are rangechecks enabled? Thats
probably the reason, but in any case, using QWord is probably the best
solution here, as the sign is not required anyway prior to conversion.

It would be nice if you could put this change into a pull request for
the repository, or at least open up an issue so I can fix this in the
repository myself later (just so I don't forgett).

----------------------------------------------------------------------
MortenB · 2021-04-14 11:03

So many things I don't know about network standards and communication
things there
Great that QWord is an acceptable solution.

As for the runtime error, yes. RangeChecks are enabled in my setup for
my project.
I am doing a bit of experimenting with types, pointers and variables,
so safer for me that way.
Working on data in many dimensions, and one error and everything goes
wrong :p

Fortunately, your websocket-solution seems to be handling everything I
need now.
What I need next is
To identify when a connection is closed from the other end.To identify
a PingTo answer a Ping with a Pong–
For my need, I can also send Pong without it being requested. I just
need to learn how to do it.
This to prevent the broadcaster from closing the connection, thinking
that it is not in use anymore.
–
Also. One last question.
Regarding speed.
How much data/text should I be able to receive per second through this
solution? Is it possible to make an estimate here, or is this
completely dependent on the Internet-connection and computer hardware?
Thanks a lot!
Morten.

----------------------------------------------------------------------
MortenB · 2021-04-14 11:34

Not sure how a pull-request works in the repository, so I made an
issue there. Such a minor one
Thanks again

----------------------------------------------------------------------
Warfley · 2021-04-14 14:12

Quote from: MortenB on April 14, 2021, 01:03:45 pmFor my need, I can
also send Pong without it being requested. I just need to learn how to
do it.
This to prevent the broadcaster from closing the connection, thinking
that it is not in use anymore.
Unsolicited pong messages are ignored according to the websocket
standard. You don't need to handle them yourself, when the
RecieveMessage function recieves a ping, it will automatically answer
with a pong. So as long as you keep receiving, it is taken care of.
Quote from: MortenB on April 14, 2021, 01:03:45 pmAlso. One last
question.
Regarding speed.
How much data/text should I be able to receive per second through this
solution? Is it possible to make an estimate here, or is this
completely dependent on the Internet-connection and computer hardware?
Can depend on both, but usually your network connection is the
limiting factor. But of course, if you have a high CPU usage, a very
old motherboard or a very fast network connection, it could be that
your network delives the data faster then the thread can read it. But
generally speaking it should be as fast as your usual download speed.
(Unless the server has a limit in, or is under heavy load, you can of
course not receive faster than the server is sending)

----------------------------------------------------------------------
MortenB · 2021-04-14 16:22

Great.
Thank you for your answers.
In regards to the Pong, I don't know much about the websocket
standards, but I have seen that Binance recommends one being sent
every half hour or so in order to ensure connection not being broken.
They also say that unsolicited are allowed.
Well. Since you say it is automatically taken care of, I will not give
that another thought.
In fact, sometimes the connection runs for 1 hour, sometimes it runs
for 18 hours. My only problem so far is to detect when it breaks so
that I can restart it, and this must be done before I start decoding
the received string of data.
At the moment, my decoder just crashes when the connection is lost,
something I expect is because the received string begins to contain
garbage at the end. :p
-
Is there a simple call I can do to check if the connection is working
properly, or should I make a check on the received string to make sure
it is Ok? Maybe just check the length, and if the length is kind of
Ok, then check the end of the string.  If it ends like a JSON-text,
then OK, and discard string and restart connection if the end of the
text is weird.

----------------------------------------------------------------------
Vaytl · 2021-07-21 10:03

Hello everyone.
Please tell me how to display a message about the disconnection with
the server? Now nothing happens, the client continues to wait, but the
connection is gone.