Reading and writing to files too slow lazarus / 64-FPC development ---------------------------------------------------------------------- semichaud1 · 2015-04-15 18:29 Hi everyone, In my program I have to read and write large files (100 MB or more) but each time it is taking too long. I have tried to use settextbuf after reset and rewrite but it makes no difference in speed. I need to use text files to store large quantities of data that my game engine uses. Could you please help? ---------------------------------------------------------------------- eny · 2015-04-15 18:42 Can you tell a bit more about how you do the reading and writing? Because with for example a TStringList reading and writing of even a 100MB file takes relatively little time. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 18:46 I am using the following commands: assign reset rewrite readln writeln and I am reading and writing ansistring and uint32 variables. Please see the attachment for a section of my program. ---------------------------------------------------------------------- BitBangerUSA · 2015-04-15 18:52 Attachments and other options - below message text-box seen when posting a reply. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 18:56 Thank you. I have attached a section of my program. ---------------------------------------------------------------------- eny · 2015-04-15 19:14 Purely looking at text I/O it should not take that long. Did a quick test reading and writing a 100MB file (1,000,000 lines) and it takes less than a second. You are also calling a couple of graphical functions (SDL_xxx and linergba) inside the loops that read the (big?) data. You could: - read and write to memory instead of diskfiles (e.g. TStringList). - cache all graphical functions and do them all áfter you have processed the text. ---------------------------------------------------------------------- BitBangerUSA · 2015-04-15 19:15 briefly looked at your attachment - but not long enough to follow what you are doing. i do note that you have only 110 elements in your 3 arrays, so - in general terms - an hour to do your task is... surprising. again, in general terms, you should look at structuring your data and the files in some way - this can certainly drastically reduce the time spent in the code to process the files. again, i didn't attempt to understand your code, so that may not be good advice. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 19:24 Thank you for your replies. The sdl function is drawing the loading bar to indicate how much is left of the process of reading and writing of files to be done. Even without the sdl command it takes a while. What I am doing essentially is this: I spent two years programming a game engine from scratch. The game engine uses data in text files. Those text files hold so much data, for example integer values for each tile of each layer in each region in the game I am making. So I programmed what I call a construction tool to add, remove and edit the values in those text files. However, the text files are so large that when I update them, which involves read and write commands, it takes a very long time. I have tried to use the settextbuf command but it made no difference in speed. ---------------------------------------------------------------------- eny · 2015-04-15 19:34 I'm guessing that you update the data time and time again? E.g. one tile changes and then you recreate the complete textfile? I would create a class that caches all that data. An interface between the physical file and the program logic. So that you read/parse the big file only once and use the data where required. Updating the data is then also done in memory and written to disk at a convenient moment, when absolutely necessary. Or you could abandon the whole textfile idea and put all data into for example a SQLite db. Gives you a lot of flexibility in storing the data in a more logical way and especially gives you the means to update only data that actually needs updating. After you've finished the 'construction' you could generate a textfile from the SQL data to be used by your game engine. Or you could use recordfiles so that you can use direct access to the data and update only the data that needs updating. (Note that when this gets more elaborate you will quickly reach the point that a SQLite db might be worthwhile.) <> By the way: goto's?? ---------------------------------------------------------------------- semichaud1 · 2015-04-15 19:58 Thank Eny for the help! So right now before updating the text file, all this data is actually loaded in arrays that are used to store the values of each tile in the selected region of the game world. What if I updated those variables only during level design and then once I am satisfied with the design of the region and where each tile is, then I updated the text file and let it take the time it needs? It should enable me not to have to wait each time I make an update to tiles in the region. I think I might do this. Why did I not think of this last night when I could not sleep trying to solve this problem in my head? ---------------------------------------------------------------------- eny · 2015-04-15 20:04 I always have the best ideas the next morning One thing to be aware of is that you lose all intermediate changes if something goes wrong (program crash or whatever). I don't know how much work it is to create your config. Another approach is that you access a textfile as a binary file: you can even use direct access to all data in a textfile. You only need to make sure that when you update data, you do not go beyond what was originally written at a certain position. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 20:15 I see. I have seen a tutorial about binary files in free pascal but I failed to see the "binary" side of it in the tutorial, lol. Maybe if you have a link to a good tutorial about this. ---------------------------------------------------------------------- BeniBela · 2015-04-15 20:28 Isn't there some buffer size you need to increase for readln/writeln? ---------------------------------------------------------------------- semichaud1 · 2015-04-15 20:36 Yes Benibela. I actually tried the settextbuf command and that made no difference. ---------------------------------------------------------------------- eny · 2015-04-15 20:40 Quote from: semichaud1 on April 15, 2015, 10:15:20 pmI see. I have seen a tutorial about binary files in free pascal but I failed to see the "binary" side of it in the tutorial, lol. Maybe if you have a link to a good tutorial about this.This is a more complex solution: you have to (binary) index each line of the file and preserve space for text and numbers when writing the file. The other alternatives are the preferred ones. Some info in the Wiki: http://wiki.freepascal.org/File_Handling_In_Pascal#Binary_files ---------------------------------------------------------------------- semichaud1 · 2015-04-15 20:46 Ok cool. I've already started implementing my solution of only updating the file once I finish design of a chosen region in the game world. However, I would really like the "increasing the buffer" solution to work as I would not need to change my code much. But I have tried that and it did not increase the speed of reading and writing. Maybe because I had to read value in char type variables, as mentioned in the tutorial, but I used ansistring and longint var instead. If I am obliged to use char var for the settextbuf to work then I don't think I'll use that solution since it will involve too many lines of codes to be rewritten. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 20:57 Just a few questions off topic: How large can a .pas file be? Is it the same limit for units too? What would limit how large a .pas can be? ---------------------------------------------------------------------- BitBangerUSA · 2015-04-15 21:06 good questions. i spent just a few minutes trying to find the answers - didn't see any. i seriously doubt that you will encounter a limit on the files mentioned - due to Lazarus/FPC... for what it's worth, i have a project with about 17 units and 25,000+ lines of code - no issues there. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 21:25 Quote from: BitBangerUSA on April 15, 2015, 11:06:23 pmgood questions. i spent just a few minutes trying to find the answers - didn't see any. i seriously doubt that you will encounter a limit on the files mentioned - due to Lazarus/FPC... for what it's worth, i have a project with about 17 units and 25,000+ lines of code - no issues there. Yeah I was asking because I never found answers to these questions. I was wandering because my project so far comes to a total of 64521 lines of codes. I have a lot of features in my game engine. Been working on it for two years. I remember when i used turbo pascal a few years back the compiler would give me an error and I had to start putting procedures in units but then it could not compile because it did not have enough memory to compile that many units. Since I switched to Free Pascal 2 years ago I have not had this issue and I'm only using 3 large units and the main program. The only "error" I had was when I had programmed a very large procedure. It did not behave as programmed (did not follow the flow of the commands) but there was no error message. ---------------------------------------------------------------------- BitBangerUSA · 2015-04-15 21:38 we can hope that a Developer (or other knowledgeable peep) will chime in here. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 21:39 Quote from: BitBangerUSA on April 15, 2015, 11:38:00 pmwe can hope that a Developer (or other knowledgeable peep) will chime in here. Lol. Would be great. ---------------------------------------------------------------------- rvk · 2015-04-15 21:55 Quote from: semichaud1 on April 15, 2015, 10:46:49 pmHowever, I would really like the "increasing the buffer" solution to work as I would not need to change my code much. But I have tried that and it did not increase the speed of reading and writing. Maybe because I had to read value in char type variables, as mentioned in the tutorial, but I used ansistring and longint var instead. No, you don't have to do read(ch); You need to specify a buffer for the read file and a buffer for the write file. FPC will also use that buffer for readln and writeln operations (flushing it to disk when necessary), not just for read(char). You can see this in the source- code at fpc\rtl\inc\text.inc (fpc_ReadLn_End, fpc_Writeln_End and SetTextBuf). Code: [Select]var   Bufin,Bufout : Array[1..60000] of byte; //....   SetTextBuf (t[1],Bufin);   SetTextBuf (t[2],Bufout); I tried some sample code with a file with 6.000.000 lines (about 100MB) doing readln and writeln in a loop just takes a few seconds (at most) to completely copy the file (even without setting a larger buffer with SetTextBuf). So I'm guessing you're doing something extra that takes this time. You could try commenting out all the writeln's to see if it speeds up your code. Otherwise you could strip this code to a working example-program and post it somewhere with a sample file so we can properly debug it (to see where the speed-problem lies). Quote from: semichaud1 on April 15, 2015, 11:25:52 pmI remember when i used turbo pascal a few years back the compiler would give me an error and I had to start putting procedures in units but then it could not compile because it did not have enough memory to compile that many units.LOL, I still get this now and then with the Borland Pascal program we still support. Borland Pascal is DOS and 16bits. The total data-segment (global variables of all units) could only be 65536 bytes. There are more limits with Borland Pascal but with FPC you won't hit the limits of 32bit (or even 64bit) that easy. (2GB for 32bit, or even 3GB with the /3GB parameter) Dividing you code into logical units will help you keep your code manageable. O yeah... Gotos ?? ---------------------------------------------------------------------- BitBangerUSA · 2015-04-15 21:59 rvk - thanks for the info. semichaud1: Turbo Pascal! wasn't that great stuff in it's day? man, i *really liked* that stuff. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 22:13 Quote from: rvk on April 15, 2015, 11:55:46 pmQuote from: semichaud1 on April 15, 2015, 10:46:49 pmHowever, I would really like the "increasing the buffer" solution to work as I would not need to change my code much. But I have tried that and it did not increase the speed of reading and writing. Maybe because I had to read value in char type variables, as mentioned in the tutorial, but I used ansistring and longint var instead. No, you don't have to do read(ch); You need to specify a buffer for the read file and a buffer for the write file. FPC will also use that buffer for readln and writeln operations (flushing it to disk when necessary), not just for read(char). You can see this in the source- code at fpc\rtl\inc\text.inc (fpc_ReadLn_End, fpc_Writeln_End and SetTextBuf). Code: [Select]var   Bufin,Bufout : Array[1..60000] of byte; //....   SetTextBuf (t[1],Bufin);   SetTextBuf (t[2],Bufout); I tried some sample code with a file with 6.000.000 lines (about 100MB) doing readln and writeln in a loop just takes a few seconds (at most) to completely copy the file (even without setting a larger buffer with SetTextBuf). So I'm guessing you're doing something extra that takes this time. You could try commenting out all the writeln's to see if it speeds up your code. Otherwise you could strip this code to a working example-program and post it somewhere with a sample file so we can properly debug it (to see where the speed-problem lies). Quote from: semichaud1 on April 15, 2015, 11:25:52 pmI remember when i used turbo pascal a few years back the compiler would give me an error and I had to start putting procedures in units but then it could not compile because it did not have enough memory to compile that many units.LOL, I still get this now and then with the Borland Pascal program we still support. Borland Pascal is DOS and 16bits. The total data-segment (global variables of all units) could only be 65536 bytes. There are more limits with Borland Pascal but with FPC you won't hit the limits of 32bit (or even 64bit) that easy. (2GB for 32bit, or even 3GB with the /3GB parameter) Dividing you code into logical units will help you keep your code manageable. O yeah... Gotos ??  Thanks! I will make a sample code tomorrow from what I have in my program. Hopefully, I can find out why this is so slow. Maybe it's because I'm reading the file line to ansistring or longint variables? I use goto because I have to go to different lines of codes in the program. The procedure is quite complex as it handles the world building function when designing the game. Also because I want to stay within the same procedure and not have to make additional procedures. This is a habit I took when programming with turbo pascal because when I made too many procedures it would run out of memory and not compile. I know I could specify the condition for exiting the repeat until loop, but I need to exit the repeat until loop at the start of the loop and not at the end if the conditions are met. Because after exiting the loop it has to read and write the lines of the text file that follow. It's due to how I have structured the data in the text files. ---------------------------------------------------------------------- semichaud1 · 2015-04-15 22:19 Quote from: BitBangerUSA on April 15, 2015, 11:59:12 pmrvk - thanks for the info. semichaud1: Turbo Pascal! wasn't that great stuff in it's day? man, i *really liked* that stuff. Yeah Turbo Pascal was the first programming language I used on the PC in 2000. I had so many projects I developed with this but unfortunately could not have fast graphics rendering. And it's only two years ago that I learned that I needed to use something like SDL to handle graphics, sound and keyboard/mouse. Wasted so many years and finally I have the knowledge and means to make a video game which is not turn-based, lol, because of limited programming language or lack of api. If I can overcome the problem I have now with slow read and write to file then I'll be all set to finish the game I am making. Hopefully, rvk can help with that after I send a proper sample of my code. ---------------------------------------------------------------------- Leledumbo · 2015-04-16 04:33 Quote from: semichaud1 on April 15, 2015, 10:57:45 pmHow large can a .pas file be? Is it the same limit for units too? What would limit how large a .pas can be? No specific limits for a filesize as it's negligible over the complexity of the source code itself. This is the documentation of compiler limits. Data type wise, fize size can be as big as High(Int64) but I believe the compiler would be dead loooooooooong before it reaches that limit due to generated data structure from the unit. Running: Code: [Select]$ find . -type f \( -name "*.pas" -or -name "*.pp" -or -name "*.inc" \) -print0 | xargs -0 ls -ShalIn fpc source dir, here are the top 10 biggest files: Quote-rw-r--r-- 1 leledumbo users 5.1M Aug 17  2014 ./packages/rtl- unicode/src/collations/collation_zh_be.inc -rw-r--r-- 1 leledumbo users 5.1M Aug 17  2014 ./packages/rtl- unicode/src/collations/collation_zh_le.inc -rw-r--r-- 1 leledumbo users 4.6M May 31  2014 ./packages/rtl- unicode/src/inc/cp936.pas -rw-r--r-- 1 leledumbo users 4.3M Aug 17  2014 ./packages/rtl- unicode/src/inc/cp949.pas -rw-r--r-- 1 leledumbo users 4.1M May 31  2014 ./packages/rtl- unicode/src/inc/cp950.pas -rw-r--r-- 1 leledumbo users 3.8M May 31  2014 ./packages/rtl- unicode/src/inc/cp932.pas -rw-r--r-- 1 leledumbo users 2.3M May 31  2014 ./packages/rtl- unicode/src/inc/ucadata_be.inc -rw-r--r-- 1 leledumbo users 2.3M May 31  2014 ./packages/rtl- unicode/src/inc/ucadata_le.inc -rw-r--r-- 1 leledumbo users 1.3M May 31  2014 ./packages/rtl- unicode/src/collations/collation_ja_be.inc -rw-r--r-- 1 leledumbo users 1.3M May 31  2014 ./packages/rtl- unicode/src/collations/collation_ja_le.inc The biggest file is 87054 LOC so your 64521 should be fine. ---------------------------------------------------------------------- JorgeAldo · 2015-04-16 04:55 are you doing read for single chars ? if yes, it IS slow from turbo pascal era and it is still slow today... you should use binary files and blockread/blockwrite or Tfilestream. ---------------------------------------------------------------------- semichaud1 · 2015-04-16 17:50 Hi everyone, please see the attached doc. Thanks! ---------------------------------------------------------------------- semichaud1 · 2015-04-16 17:50 Quote from: JorgeAldo on April 16, 2015, 06:55:15 amare you doing read for single chars ? if yes, it IS slow from turbo pascal era and it is still slow today... you should use binary files and blockread/blockwrite or Tfilestream. I'm reading and write ansistring and longint one line at a time from the file to another file. ---------------------------------------------------------------------- rvk · 2015-04-16 17:54 Quote from: semichaud1 on April 16, 2015, 07:50:15 pmHi everyone, please see the attached doc. Thanks! Do you have the possibility to provide us with a valid tf127.wor file so we can test your code? ---------------------------------------------------------------------- semichaud1 · 2015-04-16 18:14 Quote from: rvk on April 16, 2015, 07:54:54 pmQuote from: semichaud1 on April 16, 2015, 07:50:15 pmHi everyone, please see the attached doc. Thanks! Do you have the possibility to provide us with a valid tf127.wor file so we can test your code? Unfortunately the site will allow attachments up to 250 KB and tf127.wor is 103 MB ---------------------------------------------------------------------- rvk · 2015-04-16 18:17 Quote from: semichaud1 on April 16, 2015, 08:14:30 pmUnfortunately the site will allow attachments up to 250 KB and tf127.wor is 103 MB If you zip the file (with 7zip) it might even be smaller than 250Kb (due to the fact it is text with a lot of repeats). But if it's still too big... do you have dropbox (so you can share the link to the file) or another file-sharing service? ---------------------------------------------------------------------- semichaud1 · 2015-04-16 18:19 Quote from: rvk on April 16, 2015, 08:17:32 pmQuote from: semichaud1 on April 16, 2015, 08:14:30 pmUnfortunately the site will allow attachments up to 250 KB and tf127.wor is 103 MB If you zip the file (with 7zip) it might even be smaller than 250Kb (due to the fact it is text with a lot of repeats). But if it's still too big... do you have dropbox (so you can share the link to the file) or another file-sharing service? I have zipped the file. Attached. ---------------------------------------------------------------------- semichaud1 · 2015-04-16 18:24 The file takes ages to open even with Notepad ---------------------------------------------------------------------- Leledumbo · 2015-04-16 18:29 Quote from: semichaud1 on April 16, 2015, 08:24:37 pmThe file takes ages to open even with Notepad Notepad is a naive text editor. You can use text editors with on demand loading support such as whatever uses Scintilla (SciTE, Geany, Notepad++), Vim, Kate on KDE, etc. But a >100 MB text file is indeed HUGE. EDIT: Please provide a short, simple compilable example that demonstrates the behavior. Even after modifying your code to compilable one, I still get "EInOutError: Invalid input": Code: [Select]constructiontool1lb38:       readln(t[1],o[3]); // HERE so we shouldn't even start talking about speed. Apparently either the file format is wrong or the code you provided read it wrong. "EInOutError: Invalid input" will happen when the reader encounter invalid character for the expected variable's type (probably it encounters letters while o[3] is an int32). ---------------------------------------------------------------------- rvk · 2015-04-16 18:32 Yeah. Notepad (and other editors) are not really suited to open a file with 27.658.261 lines. I thought you said it had 936.000 lines: Quotethis code is reading and writing 640 * 360 * 4 lines from a text file to another but it takes nearly 1 hour to do that} But it still should take an hour. But the file you gave can't be read with the code you gave. In your code you check with the following line and break: Code: [Select]    if o[4] = o[47] then       goto constructiontool1lb38; Could you explain where o[47] is initialized? I thought the code you gave should just read the file and write a new file (with some alterations). ---------------------------------------------------------------------- BitBangerUSA · 2015-04-16 18:33 haven't tried your file, but... see attachment ---------------------------------------------------------------------- semichaud1 · 2015-04-16 18:37 Quote from: rvk on April 16, 2015, 08:32:11 pmYeah. Notepad (and other editors) are not really suited to open a file with 27.658.261 lines. I thought you said it had 936.000 lines: Quotethis code is reading and writing 640 * 360 * 4 lines from a text file to another but it takes nearly 1 hour to do that} But it still should take an hour. But the file you gave can't be read with the code you gave. In your code you check with the following line and break: Code: [Select]    if o[4] = o[47] then       goto constructiontool1lb38; Could you explain where o[47] is initialized? I thought the code you gave should just read the file and write a new file (with some alterations). o[47] holds the value of the selected world or region in the game. The code is part of a much larger unit. Basically I am trying to use text files to store data which my game engine uses to display the tiles. Is there a better way to do this than using text files? I have not studied programming and this was the logical way for me and it works but it is very slow when updating those text files. Reading only to store in variables is faster. ---------------------------------------------------------------------- semichaud1 · 2015-04-16 18:48 I'm writing a procedure to create a file with as many lines as tf127 and then read from that and copy to another file and see how long it takes. Then I'll send use the code. Bare with me because I am also doing a workout (weight lifting) session at the same time. ---------------------------------------------------------------------- rvk · 2015-04-16 18:49 Well... before diving in the o[47] issue... I tried the following code. It reproduced the tf127.wor (with all the 27.658.261 lines) in 6 seconds (on my system): Code: [Select] 6.18 seconds elapsed Just to test if it's the same on your system (elimating the source of virusscanner, defraged file, etc) could you put your tf127.wor file in C:\Temp and run the following code: Code: [Select]program test_speed2; uses Windows; var   bufin, bufout: array[1..10000] of byte;   t: array[1..10] of Text;   o: int32;   tm: Int64;   s: ansistring; begin   tm := GetTickCount;   Assign(t[1], 'c:\temp\tf127.wor');   reset(t[1]);   Assign(t[2], 'c:\temp\tf0.wor');   rewrite(t[2]);   settextbuf(t[1], bufin);   settextbuf(t[2], bufout);   repeat     readln(t[1], s);     writeln(t[2], s);   until s = 'endfile';   Close(t[1]);   Close(t[2]);   writeln((GetTickCount - tm)/1000:5:2, ' seconds elapsed');   readln; end. How fast is this on your system? (This is just to establish a base-line of how fast your system is compared to mine) (It should also show that the read and write itself isn't the problem) ---------------------------------------------------------------------- BitBangerUSA · 2015-04-16 18:50 a better way? of course. as said earlier, you need to structure your data. that's a broad statement, but the structuring will be dictated by the data *and* how you need to access it. basically, you have 2 choices - a flat text database with fixed record size, and the data 'grouped' in someway to let your program get to certain data without reading the entire file or using a traditional database. ---------------------------------------------------------------------- BitBangerUSA · 2015-04-16 19:00 huh? got the file you posted - BTW Textpad opened it in about 5 seconds on my machine. all i see is 1 8 640 360 1 4 0 0 0 0 0 0 0 0 0 0 0 0 endtotalsprites with the 1 4 0 0 ... 0 0 endtotalsprites repeated to line #27658261 it would seem you want to use a traditional database. ---------------------------------------------------------------------- rvk · 2015-04-16 19:07 It was already established on page 1 a traditional database would be preferable. At this moment we are looking at the possible speed- implication of just reading and writing this file. Not that it could be done better another way. semichaud1 just wanted to know at this moment why it takes so long (over an hour), while for me copying the file with the 27.658.261 lines takes only 6 seconds. @semichaud1: I take it you get the hint that it is better to alter this code in the future. This code is also without error-checking so any wrong character or unexpected value will completely crash your program. ---------------------------------------------------------------------- BitBangerUSA · 2015-04-16 19:08 as rvk is providing much more specific help, i'll just watch the thread. ---------------------------------------------------------------------- semichaud1 · 2015-04-16 19:15 Quote from: rvk on April 16, 2015, 09:07:24 pmIt was already established on page 1 a traditional database would be preferable. At this moment we are looking at the possible speed-implication of just reading and writing this file. Not that it could be done better another way. semichaud1 just wanted to know at this moment why it takes so long (over an hour), while for me copying the file with the 27.658.261 lines takes only 6 seconds. @semichaud1: I take it you get the hint that it is better to alter this code in the future. This code is also without error-checking so any wrong character or unexpected value will completely crash your program. rvk thanks for the help. I have attached a .txt (please change to .pas, forum does not allow attachment for this type) I made just now to replicate what I am trying to do in my program as close as possible and I get an error which I don't get in my program. Could be another issue. Sorry i need to spend more time on this. But at the moment I am working out and also I am mentally tired from work. Working all day as a credit control analyst. ---------------------------------------------------------------------- semichaud1 · 2015-04-16 19:32 Sorry guys, I'll have to work on it this weekend as it is getting late and need to work tomorrow. Thanks for the help. I really appreciate it. You are the first to help me like this of all the forums I've been to. I am just too tired from today's work. ---------------------------------------------------------------------- rvk · 2015-04-16 19:41 That's fine. Get a good nights rest. When you have the time again, please run the code I gave you to test if your system also reads and writes the 100MB files in 6 seconds. The error you get in test2....  Yeah... that's just the reason why a structured file (or database) would be more suited in this case. As I said... One unexpected value and the whole program crashes  Good night for now... (Zzzzzzzz) ---------------------------------------------------------------------- Carver413 · 2015-04-16 20:40 you should be useing index maps to store your data. that is an 8bit bitmap or larger if you need. if you have more layers then you can use one bitmap for each layer. bitmaps can be stored as binary files not text and will reduce file size and increase loading speed. on the left you see an 8 bit map and on the right you see my world. notice that I am only useing about 5 or six tiles so I still have a lot of room to grow before I would need a 16 bit map. ---------------------------------------------------------------------- semichaud1 · 2015-04-16 21:20 Guys thank you all for the great replies and help! I could not sleep unless I solved this issue. I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum! ---------------------------------------------------------------------- rvk · 2015-04-16 21:26 Quote from: semichaud1 on April 16, 2015, 11:20:18 pmI have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast.Haha, Yeah, I knew it must have been something like that. Because reading and writing alone was real fast. You won't even need a real progress-bar for 7 seconds  (B.T.W you probably won't even need the settextbuf because my guess is it will be just as fast without them) I would still advice you to look at some alternative-storage method in the future. For now it will work but the method Carver413 suggested looks much more stable. My game-programming knowledge is practically zero but my guess is that he has a lot more experience in that area. For now... you can sleep tight... Good night... ---------------------------------------------------------------------- semichaud1 · 2015-04-16 21:27 Quote from: Carver413 on April 16, 2015, 10:40:45 pmyou should be useing index maps to store your data. that is an 8bit bitmap or larger if you need. if you have more layers then you can use one bitmap for each layer. bitmaps can be stored as binary files not text and will reduce file size and increase loading speed. on the left you see an 8 bit map and on the right you see my world. notice that I am only useing about 5 or six tiles so I still have a lot of room to grow before I would need a 16 bit map. Thank you. My game engine loads bmp files into a surface array at the start of the program then values are loaded from text files into multidimensional arrays for layers. Then the layers arrays are used depending on coordinates to display the corresponding surfaces from the surface array on the screen. It's basically a tile-based engine for graphics. My engine does much more of course since I am programming a RPG and have worked on it for 2 years during my free time after work. Making the character walk around the map is super fast of course I use getticks to create proper delays for character movement and every animation in the game. The problem I was having was with updating the values of the text files during level design. But now it is solved. ---------------------------------------------------------------------- BitBangerUSA · 2015-04-16 21:45 Quote from: semichaud1 on April 16, 2015, 11:20:18 pmGuys thank you all for the great replies and help! I could not sleep unless I solved this issue. I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum! good deal. i'll assume you are now sleeping. ---------------------------------------------------------------------- eny · 2015-04-16 22:09 Quote from: semichaud1 on April 16, 2015, 11:20:18 pmGuys thank you all for the great replies and help! I could not sleep unless I solved this issue. I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum! Hm, I remember seeing that suggestion somewhere else....  Quote from: eny on April 15, 2015, 09:14:07 pmPurely looking at text I/O it should not take that long. You could: - cache all graphical functions and do them all áfter you have processed the text. ---------------------------------------------------------------------- JorgeAldo · 2015-04-17 08:01 drop the assign/reset/readln... if you need to store strings and integers, TFileStream is perfectly suited and fast. ---------------------------------------------------------------------- semichaud1 · 2015-04-17 14:59 Quote from: eny on April 17, 2015, 12:09:29 amQuote from: semichaud1 on April 16, 2015, 11:20:18 pmGuys thank you all for the great replies and help! I could not sleep unless I solved this issue. I have solved it! I removed the sdl_pollevent and linergba and sdl_flip commands from the loops and used settextbuf and now it works super fast. It reads and writes 7,372,800 lines really fast. Finally! Peace of mind. Good night and thank you again. Great forum! Hm, I remember seeing that suggestion somewhere else....  Quote from: eny on April 15, 2015, 09:14:07 pmPurely looking at text I/O it should not take that long. You could: - cache all graphical functions and do them all áfter you have processed the text. Yes, you suggested removing the graphical functions from the loop. I had done so in the past but forgot that I had not tried it with the code using settextbuf. I had codes doing it without setting the buffer. I should have paid more attention! It would have saved me a lot of time and pain. Thanks. ---------------------------------------------------------------------- semichaud1 · 2015-04-17 15:02 Quote from: JorgeAldo on April 17, 2015, 10:01:42 amdrop the assign/reset/readln... if you need to store strings and integers, TFileStream is perfectly suited and fast. I did some reading on tfilestream a few days ago. Unfortunately, this would mean changing too many lines of codes and structure. I have been programming this engine for the past two years. ---------------------------------------------------------------------- eny · 2015-04-17 17:48 Quote from: semichaud1 on April 17, 2015, 05:02:10 pmI did some reading on tfilestream a few days ago. Unfortunately, this would mean changing too many lines of codes and structure. I have been programming this engine for the past two years. Maybe something for your next rewrite; shouldn't take two years  Glad it's working now. ---------------------------------------------------------------------- semichaud1 · 2015-04-19 09:23 Just want to say thanks again everyone for your help! I'll be working on my project and hopefully, one day, I can have others playing it and liking it. I'll definitely be taking the advice for future projects. ---------------------------------------------------------------------- User137 · 2015-04-20 22:55 If you're into some advanced'ish game tile handling, there's also unit in my game engine called nxData https://code.google.com/p/nxpascal/source/browse/trunk/src/nxData.pas I've so far only made 1 (unpublished) OpenGL demo with it using Terraria graphics, where tilemap is 4200x1200 in size, and the world can be freely and quickly moved and even modified with mouse at realtime. Each tile is 16x16 in size, so it would make a 67200x19200 bitmap. At fullscreen it renders at 500 fps on my computer, when i disable default frame limiters... I never get tired praising the underlying system, which is compressing the nonvisible tile-index maps with multiple threads. Actual compressed file it makes about the whole world is 2Mb. Loading times are non-existent, fraction of a second. Of course TFileStream used.