File sync system approach lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- aradeonas · 2015-07-08 17:33 Hi, For one of my projects I want to send and receive files between server and client and for years I saw demos in Indy,Synapse ... but I wonder if any one done it in real world and have experience or suggestion that want to share with so prevent me to go wrong or reinvent wheel or engine if they are out. I want to know what is the best approach for making a sync system,what is the powerfullest library and live library,tips to know about security or even file changes recognition and upload just changed part like DropBox. I don't want all,just if you can help me in some part please do. Regards ---------------------------------------------------------------------- aradeonas · 2015-07-29 10:37 Ok.Now I want to work on file sync part and for that I need to choose a library.Indy,Synapse or ... . What you guys prefer in this case?Synapse is easy to use and I like it but Indy seems larger and more powerful but I didn't work with it. For this case that I need to send and receive files in threads with multi connections it seems I need TCP and there are demos but I want the project stable and no need to rewrite it and maybe there will be need to some p2p connections so I probably I will need to write many things myself and this is not a problem but I want to ask you what library is better in your experience and is there any demos for them in these case or not? ---------------------------------------------------------------------- jc99 · 2015-07-29 11:05 Quote from: aradeonas on July 29, 2015, 12:37:33 pmOk.Now I want to work on file sync part and for that I need to choose a library.Indy,Synapse or ... . What you guys prefer in this case?Synapse is easy to use and I like it but Indy seems larger and more powerful but I didn't work with it. For this case that I need to send and receive files in threads with multi connections it seems I need TCP and there are demos but I want the project stable and no need to rewrite it and maybe there will be need to some p2p connections so I probably I will need to write many things myself and this is not a problem but I want to ask you what library is better in your experience and is there any demos for them in these case or not? Generally use what you know and like. If you like Synapse, then use it, normally what you know best, is the best, unless you have issues with it. I used to use Indy because it came with delphi(long ago ...), so to me it would be Indy, but they now have some issues with FPC ... On the File-Sync part look at SVN or GIT how they do it, they already have a running file-syncing. Another approach is the P2P-Filetransfer (e.G: lime, torrent, donkey) (this is a file-sync of some sort. but only one-way) ---------------------------------------------------------------------- aradeonas · 2015-07-29 11:10 Im agree with you but I wanted to check before and ask others opinions not jumping in the code ---------------------------------------------------------------------- bylaardt · 2015-07-29 14:06 These files do you want synchronize comes from external pages (http or ftp)? if yes, then use synapse or indy. If the server under your responsibility and these files are not huge files, what you think about use a DBMS with external access? like mysql ou postgres without bind address? security issues can be administered by de DBMS and the files can be saved on blob fields. ---------------------------------------------------------------------- aradeonas · 2015-07-29 14:27 QuoteThese files do you want synchronize comes from external pages (http or ftp)?Files may be huge and comes from server that may be in local network or not. ---------------------------------------------------------------------- nomorelogic · 2015-07-29 16:10 as I understand, you have to realize something like rsync have you discarded the idea of using this (c) library? ---------------------------------------------------------------------- bylaardt · 2015-07-29 17:32 Quote from: aradeonas on July 29, 2015, 04:27:59 pmFiles may be huge and comes from server that may be in local network or not. "local or not" doesn't matter, but files over 2GB (2^31 -1) is can be a limit for direct access. i prefer put my files in blob fields because just a few number of users can overwrite the existing files (the file owner or authorized users), and you can create a structure of field to maintain a better control of these files, like specify the owner, version, description, search in contents, personal lists, share with information, key words, etc... with rsync is a good way to do that if system date is the only overwrite argument you need to sync. ---------------------------------------------------------------------- aradeonas · 2015-07-29 18:01 For a system like archive DB cant be good because many performance problem but in many times storing files in DB is a easy and very good way. About rsync yes it similar to it but first I saw limits in running it Windows and more important I want to have control on the system also couldn't find a good way to control it powerfully,what you think? ---------------------------------------------------------------------- bylaardt · 2015-07-29 21:06 make a pros and cons list. you need a better control? DB is my choice. you need performace? create a ftp server and access with synapse. Only if control over files is secundary. files with high Information sensitivity? my choice is Encrypt data with a server application in a protected local DB (bind address): worst option to performace. ---------------------------------------------------------------------- aradeonas · 2015-07-30 04:36 I dont know about FTP but I think a TCP client server will be a goo approach for this situation ---------------------------------------------------------------------- jc99 · 2015-07-30 07:08 Quote from: aradeonas on July 30, 2015, 06:36:28 amI dont know about FTP but I think a TCP client server will be a goo approach for this situation BTW, is only 1 client involved ? So the syncing should act as some sort of backup-mechanism ? Or do you plan to have multible clients and each can change data ? Next Question: central or decentral, central approach: more control over the whole thing, but on server- fail- no syncing possible, maybe loss of data. decentral approach: less control, but your system cannot be shut down by switching of the server. maybe slower syncing but that depends. And since it is a sync-process, only differences, files and parts of files must be synced, it can also work on a slow connection, when the initial sync-state is reached. The initiation, of cause will transfer all files, maybe compressed. (Sometimes a Harddrive shipped by (Snail-)Mail is faster than any ISP- internet-connection e.G: 6T/12h) ---------------------------------------------------------------------- gafe · 2015-07-30 07:27 I made a little command line utility as part of my "learning Pascal" experience, maybe you'll find it useful. It's another "i'm learning" file copy utility, with two options that make it different: - Copy only "changed" parts of a file. It's done reading the file in blocks (of the size you want) and storing a hash of that block in a file. Later, when copying again the file, simply checks if the block hash is the same, if not, writes that "block" in the destination file. - Read to memory a configurable (in MB) amount of files before writing them to disk. I suppose this is faster if you copy files inside the same disk. It doesn't have support for wildcards, and surely needs some love, a "overwrite" option,etc...but you can copy a c:\users folder and doesn't stop. In the end prints the files that could not be copied (if any). https://github.com/miguel-cv/dcopy/blob/master/dcopy.pas Once you read the file and have the block, you can compress it, send it over the network...what you want. ---------------------------------------------------------------------- aradeonas · 2015-07-30 09:31 @jc99 : multi client and yes like a backup and they will store in a central center. @gafe: very interesting code,Im looking at it,its a bit hard for me because of the subject and non English comment. I searched a lot but didnt find yours,You are kind with helping me and sharing it but for future please add a English description so other people can find your works easier. I will wok on it and ask you ---------------------------------------------------------------------- gafe · 2015-07-30 09:43 Sorry...i did it for me only with the purpose of learn the language, never thought of sharing it. Maybe before weekend if i have time i'll try to make/translate some comments (in English, of course). ---------------------------------------------------------------------- aradeonas · 2015-07-30 10:50 So let me know please ---------------------------------------------------------------------- aradeonas · 2015-08-12 08:23 OK I need your advice about this : I want to upload and download files and I could use Apache or others for managing server and make a http client like download and upload manager but I want to have more control in the server so I should make TCP client server if Im not wrong. So it is correct?Or Im going the wrong way?Should I make a custom protocol? If I want to explain more I should say I want to make a custom and private file  sync system to move file between server and clients and I will use normal TCP file transfer but maybe I will need UDP for p2p transfer so I want to make simple system but I dont know what is the good way for doing it. ---------------------------------------------------------------------- Graeme · 2015-08-12 08:36 Quote from: aradeonas on July 08, 2015, 07:33:30 pmFor one of my projects I want to send and receive files between server and client and for years I saw demos in Indy,Synapse ... but I wonder if any one done it in real world and have experience or suggestion Yes, the tiOPF project includes a tiFileSync utility - custom written (doesn't use existing tools like rsync) in pure Object Pascal. It allows you to sync between two local directories or between a local and remote (http) location. It was not written by myself, but rather by the creator of the tiOPF project, and he is very knowledgeable. The tiFileSync utility is also very well unit tested, so the code should be very stable. The utility was used in a commercial environment for many years already. The remote syncing support (optional) uses the HTTP protocol and the Indy components to implement it. The program is located in the "tiopf_apps" repository, in the tiFileSync directory:   http://sourceforge.net/p/tiopf/tiopf_apps/ci/master/tree/ ---------------------------------------------------------------------- aradeonas · 2015-08-12 08:40 Can I use it for my custom usage like ability to use extra protocols like FTP o SMB if I want? what is the best way to doing it? Take https://owncloud.com as an example,its a very much complicated version of what I want to do but take it as an example but I want to do it in pascal. I know its much to ask but I always wanted to learn about this and I like to make it in Pascal. ---------------------------------------------------------------------- Thaddy · 2015-08-12 08:49 A time stamp, a date a cache and a hash, that's the way to call sync. ---------------------------------------------------------------------- aradeonas · 2015-08-12 08:51 Quote from: Thaddy on August 12, 2015, 10:49:58 amA time stamp, a date a cache and a hash, that's the way to call sync. good one and true ! Graeme its reminds me synopse . Are they doing the same job? Can you explain it more? ---------------------------------------------------------------------- Graeme · 2015-08-12 09:24 Quote from: aradeonas on August 12, 2015, 10:40:19 amCan I use it for my custom usage like ability to use extra protocols like FTP o SMB if I want? what is the best way to doing it? As with all things tiOPF - the design is very good and very flexible. tiOPF and associated tools always use the power of design patterns to make the software more flexible. So yes, you will be able to easily add other protocol options like FTP etc to tiFileSync. Even using Synapse instead of Indy will be easy. ---------------------------------------------------------------------- Graeme · 2015-08-12 09:34 Quote from: aradeonas on August 12, 2015, 10:51:34 amGraeme its reminds me synopse . Are they doing the same job? I've never used mORMot or Synopse, so can't really compare it to tiOPF. All I can say is that tiOPF is the oldest (been around since before 1999) and most stable Object Persistence Framework for Delphi and FPC I know of. I maintain the servers that run 1750+ unit test for tiOPF every 3 hours, testing everything in the framework and various text and database backends. It has been in production use for 16+ years. I've written most of my projects in the last 15 years using it, and development still continues today. To find out more about tiOPF please visit http://www.tiopf.com. ---------------------------------------------------------------------- aradeonas · 2015-08-12 11:17 Thank you.I will work on it and ask about it.