[SOLVED] This is driving me mad: "Inconsistency detected by ld.so" lazarus / 16-Databases ---------------------------------------------------------------------- Zvoni · 2019-03-26 05:11 Hi folks, close to deleting FPC/Lazarus. Ubuntu18.10-64bit/FPC304/Laz200 Library with exported functions. First call to the library creates a Thread, which takes over as MainThread (because of CheckSynchronize). This FakeMainthread creates a/many childthreads. Everything woking so far. The moment i add DB-Components to the Childthreads, everything goes south. If i use MySQL in the TSQLConnector, it works (with one thread! Many threads it still goes haywire) If i use SQLite3 i get the Error-MEssage "Inconsistency detected by ld.so: dl-minimal.c: 126: realloc: Assertion `ptr == alloc_last_block' failed!" I found exactly one thread in the forum having the error-message. And the discussion there didn't help. cthreads is the first Unit in the libraray.lpr and i don't use cmem, because that one doesn't work for me (heaptrc always saying no Memory leaks), and i get a lot of corrupted memory from cmem. Help? EDIT: I forgot: The error occurs after connection has been established (connection.connected returns true), on the first try to grab any information from the database (Query.Open, Connection.GetTableNames etc.) EDIT2: Before anyone asks: YEs, i have successfully worked with SQLite under FPC304, so i don't understand what's happening here, since i don't do anything different than before EDIT3: And yes: Each thread has its own DB-Conn, Trans and Query! I'm not mixing them, and the threads/db-components don't have to talk with each other. ---------------------------------------------------------------------- Thaddy · 2019-03-26 06:38 Quote from: Zvoni on March 26, 2019, 06:11:58 amand i don't use cmem, because that one doesn't work for me (heaptrc always saying no Memory leaks), and i get a lot of corrupted memory from cmem. That is because you overlooked  that heaptrc IS a memory manager by itself, so if you hook cmem, heaptrc won't work anymore. You can safely use heaptrc and if no leaks are detected change it to cmem. That's what heaptrc is designed for. Just to be sure: you should let the compiler insert heaptrc and not put it in any uses clause. The above problems can be wholly caused by mixing two different memory managers, so try first heaptrc without cmem and then with just cmem and without the heaptrc option set. If there is still memory corruption we need to look further. The reason I suspect this is the case is the following: When the compiler inserts heaptrc it does so *before* anything else is done and heaptrc already allocates memory by itself to be able to register all memory allocations/de-allocations. (that's why it is done by the compiler! and not through a uses clause) That memory is lost when you subsequently set a different memory manager: acute memory corruption. ---------------------------------------------------------------------- Zvoni · 2019-03-26 08:07 Quote from: Thaddy on March 26, 2019, 07:38:21 amQuote from: Zvoni on March 26, 2019, 06:11:58 amand i don't use cmem, because that one doesn't work for me (heaptrc always saying no Memory leaks), and i get a lot of corrupted memory from cmem. That is because you overlooked  that heaptrc IS a memory manager by itself, so if you hook cmem, heaptrc won't work anymore. You can safely use heaptrc and if no leaks are detected change it to cmem. That's what heaptrc is designed for. Just to be sure: you should let the compiler insert heaptrc and not put it in any uses clause.No heaptrc in Uses-clause. I added it in Project-Options-Debugger (Dwarf2) And i've removed cmem from uses-clause a "long time ago", but i admit: didn't know that you can't/shouldn't use those two together. Maybe put that on the HeapTrc-Wiki: DON'T USE IT WITH ANY OTHER MEMORY-MANAGER. QuoteThe above problems can be wholly caused by mixing two different memory managers, so try first heaptrc without cmem and then with just cmem and without the heaptrc option set. If there is still memory corruption we need to look further. The reason I suspect this is the case is the following: When the compiler inserts heaptrc it does so *before* anything else is done and heaptrc already allocates memory by itself to be able to register all memory allocations/de-allocations. (that's why it is done by the compiler! and not through a uses clause) That memory is lost when you subsequently set a different memory manager: acute memory corruption. EDIT: Another thing i don't understand: The Error-Message i'm getting implies a problem with the linker. But both, the library and the Test-Program compile without hitches, and the linker throws the error, when it reaches the point in the code as described above. Funny thing: only with SQLite. A single thread accessing MySQl works (2/more threads accessing MySQL is another can of worms). The library is accessed during runtime via the exported functions (lib is not statically linked into the main program) ---------------------------------------------------------------------- Cyrax · 2019-03-26 10:39 Open a terminal; issue these commands and check how high is the stack size set in Ubuntu system. Quote#soft limits for a process/thread ulimit -a #hard limits for a process/thread ulimit -Ha Then set the soft limit of a stack size for reasonable size eg. 512 (ulimit -s 512) and try to re-execute your program. ---------------------------------------------------------------------- Thaddy · 2019-03-26 10:48 I have tried to replicate your problem on win64 and Raspbian, but I used 3.2.0 and 3.3.1 (from today). I have no problem using sqlite3 in any way, whatever memory manager I use. Note that the Sqlite3 bindings in 3.2.0 and trunk have changed a bit to support threading in a better way. ---------------------------------------------------------------------- Zvoni · 2019-03-26 11:12 Quote from: Thaddy on March 26, 2019, 11:48:36 amI have tried to replicate your problem on win64 and Raspbian, but I used 3.2.0 and 3.3.1 (from today). I have no problem using sqlite3 in any way, whatever memory manager I use. Note that the Sqlite3 bindings in 3.2.0 and trunk have changed a bit to support threading in a better way. Thaddy, thx for that information. I'll postpone on SQLite-use until second half of the year then, when FPC320 becomes officially available (not familiar with building from svn/trunk/wherever) ---------------------------------------------------------------------- Zvoni · 2019-03-26 11:14 Quote from: Cyrax on March 26, 2019, 11:39:21 amOpen a terminal; issue these commands and check how high is the stack size set in Ubuntu system. Quote#soft limits for a process/thread ulimit -a #hard limits for a process/thread ulimit -Ha Then set the soft limit of a stack size for reasonable size eg. 512 (ulimit -s 512) and try to re-execute your program. thx, will try that and report back ---------------------------------------------------------------------- Cyrax · 2019-03-26 12:43 Quote from: Zvoni on March 26, 2019, 12:12:29 pmQuote from: Thaddy on March 26, 2019, 11:48:36 amI have tried to replicate your problem on win64 and Raspbian, but I used 3.2.0 and 3.3.1 (from today). I have no problem using sqlite3 in any way, whatever memory manager I use. Note that the Sqlite3 bindings in 3.2.0 and trunk have changed a bit to support threading in a better way. Thaddy, thx for that information. I'll postpone on SQLite-use until second half of the year then, when FPC320 becomes officially available (not familiar with building from svn/trunk/wherever) You can use fpcupdeluxe for that : http://wiki.lazarus.freepascal.org/fpcupdeluxe ---------------------------------------------------------------------- Zvoni · 2019-03-26 17:29 Quote from: Zvoni on March 26, 2019, 12:14:28 pmQuote from: Cyrax on March 26, 2019, 11:39:21 amOpen a terminal; issue these commands and check how high is the stack size set in Ubuntu system. Quote#soft limits for a process/thread ulimit -a #hard limits for a process/thread ulimit -Ha Then set the soft limit of a stack size for reasonable size eg. 512 (ulimit -s 512) and try to re-execute your program. thx, will try that and report back My result on my machine Code: [Select]zvoni@ZvoniLinux:~$ ulimit -a core file size          (blocks, -c) 0 data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 31133 max locked memory       (kbytes, -l) 16384 max memory size         (kbytes, -m) unlimited open files                      (-n) 1024 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) 8192 cpu time               (seconds, -t) unlimited max user processes              (-u) 31133 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited zvoni@ZvoniLinux:~$ ulimit -Ha core file size          (blocks, -c) unlimited data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 31133 max locked memory       (kbytes, -l) 16384 max memory size         (kbytes, -m) unlimited open files                      (-n) 4096 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) unlimited cpu time               (seconds, -t) unlimited max user processes              (-u) 31133 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited ---------------------------------------------------------------------- Thaddy · 2019-03-27 09:20 Quote from: Zvoni on March 26, 2019, 09:07:04 amMaybe put that on the HeapTrc-Wiki: DON'T USE IT WITH ANY OTHER MEMORY-MANAGER. FYI: I just adapted the wiki and added a warning, but also a note how to handle alternative memory managers and heaptrc in the same codebase: http://wiki.freepascal.org/heaptrc http://wiki.freepascal.org/heaptrc#Why_heaptrc_should_not_be_added_to_ the_uses_clause_manually ---------------------------------------------------------------------- Cyrax · 2019-03-27 09:39 Quote from: Zvoni on March 26, 2019, 06:29:57 pmQuote from: Zvoni on March 26, 2019, 12:14:28 pmQuote from: Cyrax on March 26, 2019, 11:39:21 amOpen a terminal; issue these commands and check how high is the stack size set in Ubuntu system. Quote#soft limits for a process/thread ulimit -a #hard limits for a process/thread ulimit -Ha Then set the soft limit of a stack size for reasonable size eg. 512 (ulimit -s 512) and try to re-execute your program. thx, will try that and report back My result on my machine Code: [Select]zvoni@ZvoniLinux:~$ ulimit -a core file size          (blocks, -c) 0 data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 31133 max locked memory       (kbytes, -l) 16384 max memory size         (kbytes, -m) unlimited open files                      (-n) 1024 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) 8192 cpu time               (seconds, -t) unlimited max user processes              (-u) 31133 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited zvoni@ZvoniLinux:~$ ulimit -Ha core file size          (blocks, -c) unlimited data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 31133 max locked memory       (kbytes, -l) 16384 max memory size         (kbytes, -m) unlimited open files                      (-n) 4096 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) unlimited cpu time               (seconds, -t) unlimited max user processes              (-u) 31133 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited What does ulimit -Ha say? ---------------------------------------------------------------------- Zvoni · 2019-03-27 10:14 Quote from: Cyrax on March 27, 2019, 10:39:15 am Code: [Select]zvoni@ZvoniLinux:~$ ulimit -a core file size          (blocks, -c) 0 data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 31133 max locked memory       (kbytes, -l) 16384 max memory size         (kbytes, -m) unlimited open files                      (-n) 1024 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) 8192 cpu time               (seconds, -t) unlimited max user processes              (-u) 31133 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited zvoni@ZvoniLinux:~$ ulimit -Ha Code: [Select]core file size          (blocks, -c) unlimited data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 31133 max locked memory       (kbytes, -l) 16384 max memory size         (kbytes, -m) unlimited open files                      (-n) 4096 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) unlimited cpu time               (seconds, -t) unlimited max user processes              (-u) 31133 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited What does ulimit -Ha say? huh? ---------------------------------------------------------------------- Zvoni · 2019-03-27 10:14 Quote from: Thaddy on March 27, 2019, 10:20:37 amQuote from: Zvoni on March 26, 2019, 09:07:04 amMaybe put that on the HeapTrc-Wiki: DON'T USE IT WITH ANY OTHER MEMORY-MANAGER. FYI: I just adapted the wiki and added a warning, but also a note how to handle alternative memory managers and heaptrc in the same codebase: http://wiki.freepascal.org/heaptrc http://wiki.freepascal.org/heaptrc#Why_heaptrc_should_not_be_added_to_ the_uses_clause_manually Thaddy, thx +1 ---------------------------------------------------------------------- Cyrax · 2019-03-27 11:10 Quote from: Zvoni on March 27, 2019, 11:14:04 amQuote from: Cyrax on March 27, 2019, 10:39:15 am Code: [Select]zvoni@ZvoniLinux:~$ ulimit -a core file size          (blocks, -c) 0 data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 31133 max locked memory       (kbytes, -l) 16384 max memory size         (kbytes, -m) unlimited open files                      (-n) 1024 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) 8192 cpu time               (seconds, -t) unlimited max user processes              (-u) 31133 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited zvoni@ZvoniLinux:~$ ulimit -Ha Code: [Select]core file size          (blocks, -c) unlimited data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 31133 max locked memory       (kbytes, -l) 16384 max memory size         (kbytes, -m) unlimited open files                      (-n) 4096 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) unlimited cpu time               (seconds, -t) unlimited max user processes              (-u) 31133 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited What does ulimit -Ha say? huh? Oh sorry, I didn't see the ulimit -Ha part from the quote text field.  It seems there is no limits. Please open a terminal in your program directory, execute ulimit -s 512 and then execute your program and see if there is errors or program misbehaves . EDIT : Oh, there is soft limit: Quotestack size              (kbytes, -s) 8192 ---------------------------------------------------------------------- Zvoni · 2019-03-27 11:26 Quote from: Cyrax on March 27, 2019, 12:10:23 pm Oh sorry, I didn't see the ulimit -Ha part from the quote text field.  It seems there is no limits. Please open a terminal in your program directory, execute ulimit -s 512 and then execute your program and see if there is errors or program misbehaves . Definitely no expert here, but why should i lower my Stack-size from 8196 KB to 512? (ulimit -s is for the "soft"-limit?!) ---------------------------------------------------------------------- Cyrax · 2019-03-27 11:41 Quote from: Zvoni on March 27, 2019, 12:26:15 pmQuote from: Cyrax on March 27, 2019, 12:10:23 pm Oh sorry, I didn't see the ulimit -Ha part from the quote text field.  It seems there is no limits. Please open a terminal in your program directory, execute ulimit -s 512 and then execute your program and see if there is errors or program misbehaves . Definitely no expert here, but why should i lower my Stack-size from 8196 KB to 512? (ulimit -s is for the "soft"-limit?!) Just for testing purposes. Every created thread (yes, every thread too and there can be several hundred of them running) and process allocates the default 8196 KB of memory.  And if there is about 1024 threads running then kernel will allocate about 8 192 MB ( that is 8 GB!)  for them. It is bit too much. So lowering it to between 512KB - 1024KB should suffice. ---------------------------------------------------------------------- Cyrax · 2019-03-27 11:45 Info about on this : https://unix.stackexchange.com/questions/127602/default-stack-size- for-pthreads ---------------------------------------------------------------------- engkin · 2019-03-27 13:33 Don't let us guess, provide a sample project that shows the problem. ld.so is the dynamic linker. Works when you run your app. ---------------------------------------------------------------------- Zvoni · 2019-03-27 15:54 Quote from: Cyrax on March 27, 2019, 12:45:50 pmInfo about on this : https://unix.stackexchange.com/questions/127602/default-stack-size- for-pthreads Thx for the link. Learned something new ---------------------------------------------------------------------- Zvoni · 2019-03-27 15:57 Quote from: engkin on March 27, 2019, 02:33:14 pmDon't let us guess, provide a sample project that shows the problem. ld.so is the dynamic linker. Works when you run your app. The problem seems to be the sqlite-connection-object in context of running the connection in its own thread. I don‘t have the error with MySQL, and as Thaddy mentioned, they changed some sqlite-bindings in trunk, where it doesn‘t report the error, while i‘m on fpc304 ---------------------------------------------------------------------- Zvoni · 2019-03-30 09:36 @Thaddy By any chance: In trunk, are you using sqlite_open_v2 for InternalConnect? I‘ve looked through the sourcecode, and found sqlite_open being used, and the docs for sqlite say, in that case sqlite is run with compiletime-options regarding multiple threads. The default compile-options for distro-binaries is mode 1 (Serialized?), so right now i don‘t have a clue if it‘s my sqlite-lib being in singlethread-mode, or what you mentioned. I‘m going to setup a small test, if the error occurs in a naked, stripped down, code ---------------------------------------------------------------------- Thaddy · 2019-03-30 09:52 Quote from: Zvoni on March 30, 2019, 10:36:10 am@Thaddy By any chance: In trunk, are you using sqlite_open_v2 for InternalConnect? I‘ve looked through the sourcecode, and found sqlite_open being used, and the docs for sqlite say, in that case sqlite is run with compiletime-options regarding multiple threads. The default compile-options for distro-binaries is mode 1 (Serialized?), so right now i don‘t have a clue if it‘s my sqlite-lib being in singlethread-mode, or what you mentioned. I‘m going to setup a small test, if the error occurs in a naked, stripped down, code Yes. The Sqlite3 component for SqlDb is adapted to use sqlite_open_v2 by default on my request. Target release is 3.2.0 . See https://bugs.freepascal.org/view.php?id=34278 which also documents briefly the new options available. (That change definitely made Sqlite3 more robust., on my side all kind of quirks disappeared after the change!) In your case I would suggest a defensive approach and use: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i OpenFlags := [sofReadWrite,sofCreate,sofFullMutex]; // and maybe one of the caching strategiesUnless you run into performance issues.  See the Sqlite3 documentation, which you can now use a bit better with FPC. You also still have to keep in mind Cyrax remark about lowering the stack size on some platforms. ---------------------------------------------------------------------- Zvoni · 2019-03-30 10:58 Quote from: Thaddy on March 30, 2019, 10:52:49 amYou also still have to keep in mind Cyrax remark about lowering the stack size on some platforms. About that: Any pointers how to do that? Still fairly new to everything. At a guess: there is somewhere in Project-Options a Flag i can set the stacksize.... EDIT. Found it. The "-Cs"-Option? Value is what? in Kb? Like "-Cs 512" is setting Stacksize to 512 Kb? ---------------------------------------------------------------------- Thaddy · 2019-03-30 11:05 Quote from: Zvoni on March 30, 2019, 11:58:30 amQuote from: Thaddy on March 30, 2019, 10:52:49 amYou also still have to keep in mind Cyrax remark about lowering the stack size on some platforms. About that: Any pointers how to do that? Still fairly new to everything. At a guess: there is somewhere in Project-Options a Flag i can set the stacksize.... EDIT. Found it. The "-Cs"-Option? Value is what? in Kb? Like "-Cs 512" is setting Stacksize to 512 Kb? You can also set {$MINSTACKSIZE number} and  {$MAXSTACKSIZE number} in your code. The minimum value for {$maxstacksize} is 2048, though, you can't set it to 512, but 2048 should suffice in most cases. Note that this goes for threading with TThread. If you want your threads to have a different or smaller stacksize, you should not use TThread but any of the beginthread methods: https://www.freepascal.org/docs-html/rtl/system/beginthread.html I rarely use TThread in scenario's such as yours, because beginthread is more comfortable and has more control. ---------------------------------------------------------------------- Cyrax · 2019-03-30 11:19 Quote from: Zvoni on March 30, 2019, 11:58:30 amQuote from: Thaddy on March 30, 2019, 10:52:49 amYou also still have to keep in mind Cyrax remark about lowering the stack size on some platforms. About that: Any pointers how to do that? Still fairly new to everything. At a guess: there is somewhere in Project-Options a Flag i can set the stacksize.... EDIT. Found it. The "-Cs"-Option? Value is what? in Kb? Like "-Cs 512" is setting Stacksize to 512 Kb? Open terminal window and execute command ulimit -s 512 and after executing that command, execute your program. How to open terminal window : https://askubuntu.com/questions/183775/how-do-i-open-a-terminal ---------------------------------------------------------------------- Zvoni · 2019-03-30 12:06 Quote from: Zvoni on March 30, 2019, 10:36:10 amI‘m going to setup a small test, if the error occurs in a naked, stripped down, code Oh, come on. I just set up a barebone test in the scenario i have (albeit not in a library, but a console-program), and sqlite doesn't throw any errors in the threads. Create FakeMainThread, take over as MainThread, FakeMainThread creates Childthreads, which itself create the TSQL- Trinity with Type 'SQLite3', connect to test-database, and no error is thrown by any of the TSQL- Objects accessing data of the DB *sigh* Back to square one, and off i go bug-hunting....... [CLOSED] ---------------------------------------------------------------------- Zvoni · 2019-03-30 12:46 Quote from: Zvoni on March 30, 2019, 11:58:30 amQuote from: Thaddy on March 30, 2019, 10:52:49 amYou also still have to keep in mind Cyrax remark about lowering the stack size on some platforms. About that: Any pointers how to do that? Still fairly new to everything. At a guess: there is somewhere in Project-Options a Flag i can set the stacksize.... EDIT. Found it. The "-Cs"-Option? Value is what? in Kb? Like "-Cs 512" is setting Stacksize to 512 Kb? How curious. Setting Stacksize in Project-Options to anything other than 0 results: 512 --> won't complile. No messages, no nothing 1024 (and higher) --> Stackoverflow Error Switching off "-Ct" 512 --> won't complile. No messages, no nothing 1024 (and higher) --> seems (!) to work ---------------------------------------------------------------------- engkin · 2019-03-30 14:15 It is a parameter you pass to the thread constructor. ---------------------------------------------------------------------- Zvoni · 2019-03-30 17:56 Quote from: engkin on March 30, 2019, 03:15:44 pmIt is a parameter you pass to the thread constructor. Ahhh, OK. Have seen it, looked it up, but since no example ever shows its use, i ignored it. EDIT: Nope, whatever i pass to the inherited constructor (512,1024,2048,4096) results in an Exception "Could not Create Thread" The Stacksize in Project-options was set to 0 for this test. EDIT2: Writeln('DefaultStackSize = '+DefaultStackSize.ToString);    --> 4194304 What's that? Apples? Bytes? Beers? EDIT3: *GNARF* passing 524288 works (a.k.a 512kb) ---------------------------------------------------------------------- Zvoni · 2019-03-31 12:21 OK, we can close this Thread. I've rebuild my code-base in a new project, and nowhere the afore mentioned error occurs. Seems to have been a bug in my code (no idea where)