Paperless-ngx for 10TB of data reddit / selfhosted ---------------------------------------------------------------------- lambadime · 2023-11-08 09:30 I have proposed in my office to use paperless-ngx for storing and sharing all scanned documents. It is estimated that around 10TB of data will be generated post scanning of all docs in the office., each doc ranging from 10mb to 500mb with min 30 pages to 800 pages. I was so confident with the solution and was able to convenience the management to go with the solution. But now i am noticing that paperless-ngx is taking way tooo long time to add documents with more pages or more size. eg: 30mb of file with 600 pages OCR time is more than a 100mb file with 3 pages. What is the logic and why it is taking time. Does paperless-ngx really support 10TB of data storage? Need suggestions. ---------------------------------------------------------------------- OppositeFisherman89 · 2023-11-08 09:45 Have you considered setting PAPERLESS_OCR_PAGES to 1, so that paperless will only OCR the first page of your documents? ---------------------------------------------------------------------- Couch941 · 2023-11-08 09:46 >Why do 600 pages take longer than 3 pages Because it's more pages? What answer do you expect. ​ >Does paperless-ngx really support 10TB of data storage? Yes, but I am not sure if it was made for your sort of application ---------------------------------------------------------------------- RedditSlayer2020 · 2023-11-08 09:56 Buy an enterprise solution. If your management doesn't have a budget for that you haven't convinced anyone. Stop begging selfhosted communities for cheap solutions. ---------------------------------------------------------------------- lambadime · 2023-11-08 10:07 I had an understanding that the ocr time taken is based on the size of the document and not strictly based on number of pages. is it like that or my configuration is somewhere wrong. ---------------------------------------------------------------------- lambadime · 2023-11-08 10:07 yup , did that too. still looks like its taking all the pages for OCRing ---------------------------------------------------------------------- Couch941 · 2023-11-08 10:08 Why would it be based on the size of the document? The amount of pages is all that matters. It just looks at the page, tries finding/reading the text and that's it. It doesn't matter if you embed a 100MB image in the page, as it's still just one page. ---------------------------------------------------------------------- lambadime · 2023-11-08 10:09 Boss its not about begging. Its the vision towards using only opensource solutions and developing specialisations on it. ---------------------------------------------------------------------- RedditSlayer2020 · 2023-11-08 10:18 Sorry to come across salty but I'm just sick of businesses coming here looking for cheap solutions instead of buying an enterprise solution. That they don't even try to hide/disguise it anymore is kind of infuriating. Having clownflare chills in here is already annoying enough. Nothing personal though. ---------------------------------------------------------------------- lambadime · 2023-11-08 10:26 got it . ---------------------------------------------------------------------- louisj · 2023-11-08 10:27 Another way to phrase it is that even with open source, you still need enterprise level support ---------------------------------------------------------------------- Verolee · 2023-11-08 10:37 I’m going to be really helpful right now. There’s another one, that I thought had more complex features, but I forgot the name. It’s like Aztec or Mesa or something like that. ---------------------------------------------------------------------- lambadime · 2023-11-08 10:42 >Aztec couldnt find one with that name..whats that ---------------------------------------------------------------------- lighthawk16 · 2023-11-08 10:43 The size (resolution) of the document does matter. Higher resolutions will take longer for the OCR to scan. Even if a minute amount. ---------------------------------------------------------------------- Verolee · 2023-11-08 10:47 I found it!! [Mayan](https://www.mayan-edms.com) ---------------------------------------------------------------------- benutzername63532 · 2023-11-08 11:19 The in App Preview of the File wasnt working for me so i decides on paperless ---------------------------------------------------------------------- djc_tech · 2023-11-08 11:41 Try mayanedms. It’s geared more towards enterprise/business and offers support. ---------------------------------------------------------------------- KyroPaul · 2023-11-08 11:46 I love paperless and tell everyone who self hosts to use it, but I would never put it on my corporate network. It just doesn't seem like the right tool for that (your company structure might vary and it makes sense). If you need something that can parse documents, run ocr and move files into folders you should check out a more purpose built tool like chronoscan (not open source as far as I know but it is self hosted and a lot can be built on-top of it), you could also look at Alfresco (open-source) if you want a doc repo after you scan the document and decide what employees/departments want access to it. ---------------------------------------------------------------------- Verolee · 2023-11-08 11:54 Yea it was a bitch to set up ---------------------------------------------------------------------- Verolee · 2023-11-08 11:55 Yea, thought that 10tb would be considered that ---------------------------------------------------------------------- rmusic10891 · 2023-11-08 12:44 I used to oversee the team that ran the enterprise document capture solution for a really large bank. We mostly talked about the number of pages we needed to ingest, and would talk a little about DPI (mostly for accuracy reasons). With our enterprise solution we could OCR 1 page per second per CPU core. We took in about a million pages a day. Assuming a standard 300DPI page was about 40KB you're talking about 40GB a day... on an enterprise solution. Even scaling up for modern hardware I imagine 10TB would take months. ---------------------------------------------------------------------- formkiq · 2023-11-08 13:14 If you don't mind storing in Amazon S3 (which has its benefits), you could also look into our scalable open source solution, FormKiQ Core ([https://github.com/formkiq](https://github.com/formkiq)). It's not self-hosted, of course, but may be a better fit for the enterprise. ---------------------------------------------------------------------- thekrautboy · 2023-11-08 13:54 Very much agree. ---------------------------------------------------------------------- LastTreestar · 2023-11-08 14:12 Stop being selfish, and think of the value you are bringing to the shareholders. ---------------------------------------------------------------------- MyTechAccount90210 · 2023-11-08 15:07 In addition it's waayy more complicated too. I tried it for home, being oss and all, and it was ridiculous. I couldn't even get it fully deployed properly. ---------------------------------------------------------------------- its_me_mario9 · 2023-11-08 15:28 One thing you haven’t mentioned is the specs of the system you’re using to digest said documents. Have you considered that it just may not be powerful enough? ---------------------------------------------------------------------- user798123 · 2023-11-08 15:29 This is fascinating! What did your team do to scale this? Was it matter of throwing more hardware at it? And if so, what sort of hardware did the system use? ---------------------------------------------------------------------- d-cent · 2023-11-08 15:42 This is very fascinating and I hate that the people downvoted the OP's comment before it, because so many people won't see your comment. This is why downvoting brigades are meant to curtail hate speech, not when someone is commenting an honest question that they don't know much about. ---------------------------------------------------------------------- rmusic10891 · 2023-11-08 15:58 This was 5 years ago at this point so our ability to scale at that time was pretty limited. We actually two environments we would use for load balancing. Our main indexing software was EMC Captiva (this is probably the most popular enterprise solution but it’s changed hands a couple times and I think it now lives on as OpenText Intelligent Capture and Core Capture and can be deployed to multi-tenant SaaS). We used this solution for automated document classification and text extraction for process automation. Example would be extracting text off a standardized URLA form and sending it off to one of our mortgage origination or servicing applications. We also had bulk ingestion capabilities for bulk loading files that either followed a standard naming convention or had appropriate metadata to index the document to IBM FileNet. ---------------------------------------------------------------------- Nerdlinger42 · 2023-11-08 16:00 How do people overlook this? I'm a system admin. I would have left IT if there wasn't vendor support for most of the apps I manage. At some point you just throw the hands up and call the vendor. ---------------------------------------------------------------------- Significant-Neat7754 · 2023-11-08 16:21 Could you let me know what database one should use for data that huge? I'm guessing sqlite won't work. ---------------------------------------------------------------------- majesticPolishJew · 2023-11-08 16:43 This is an awesome answer ---------------------------------------------------------------------- xCharg · 2023-11-08 16:44 Experienced enterpriser :) ---------------------------------------------------------------------- leoklaus · 2023-11-08 18:54 I’ve set up a test instance of ngx with about 3000 random pdf documents from 1 to ~150 pages per document. Apart from consumption taking extremely long, the performance of the interface itself was abysmal (running on a 2C/8GB Ampere ARM VM) it was so slow that api requests straight up timed out. No matter how fast your server and or database is, there’s no way Paperless can handle 10TB of documents. ---------------------------------------------------------------------- HolyPally94 · 2023-11-08 19:04 Just a hint, it is clear that OCR a document that has 600 pages takes longer than a three pager. This is because every page is OCR'd separately. Thus, it is helpful to know that there are options to increase the amount of workers that paperless uses. With the default I think it is set to 1 which means one page after the other. If you've got a powerful server, you can increase this to process multiple pages at once. Also, there is another option to configure how many new documents will be processed in parallel. In general, multiplying these two parameter values should not exceed your CPU count. ---------------------------------------------------------------------- HolyPally94 · 2023-11-08 19:08 Just a hint, did you try to increase the number of workers? There are options to increase the 'frontend workers' (don't know exactly their terminology). This should increase the performance of the web interface. With my current instance, I am about to hit the 2000 documents mark and it is extremely fast. Though I increased the frontend workers and also the backend workers and parallel document cosumption. ---------------------------------------------------------------------- leoklaus · 2023-11-08 19:11 Do you have hundreds of documents with tens of pages each? I’ve since deleted this test instance and set it up on the same machine with the same configuration but a dataset of just single-page pdfs. I’m now at ~5000 documents with very good performance. I’m pretty sure the main issue was the (for database terms) very large „content“ field. ---------------------------------------------------------------------- HolyPally94 · 2023-11-08 19:14 For sure not, most of my documents are 2-3 pages, some (roughly 100) are 10-20 pages long. Just wanted to mention [the option](https://docs.paperless- ngx.com/configuration/#PAPERLESS_WEBSERVER_WORKERS) :) ---------------------------------------------------------------------- HolyPally94 · 2023-11-08 19:15 I mean [this option](https://docs.paperless- ngx.com/configuration/#PAPERLESS_TASK_WORKERS) and the following one. ---------------------------------------------------------------------- leoklaus · 2023-11-08 19:16 Sure, thank you! ---------------------------------------------------------------------- dutchie_ok · 2023-11-08 20:40 I don't know your setup but I'd start with profiling used DB. Maybe changing some parameters, ie memory consumption would be enough? ---------------------------------------------------------------------- dutchie_ok · 2023-11-08 20:40 I don't know your setup but I'd start with profiling used DB. Maybe changing some parameters, ie memory consumption would be enough? ---------------------------------------------------------------------- sjveivdn · 2023-11-08 21:18 Did you use the docker container? ---------------------------------------------------------------------- hyperactive2 · 2023-11-08 21:59 As previously stated, you might need an enterprise solution. You should spend some time trying to distribute the work load across multiple hosts, I don't know if anyone has ever tried it with paperless-ngx. At the very least, setup a beefy database server and a SUPER BEEFY webserver. For single machine performance, take a look at [https://docs.paperless-ngx.com/configuration/#PAPERLESS\_THREADS\_PER \_WORKER](https://docs.paperless- ngx.com/configuration/#paperless_threads_per_worker) and [https://docs.paperless-ngx.com/configuration/#PAPERLESS\_TASK\_WORKER S](https://docs.paperless- ngx.com/configuration/#paperless_task_workers) I just did a test on my 8C with 50 PDFs and adding a second worker with 4 threads each noticeably reduced the import time. Lastly, I would suggest limiting the OCR to 2-3 pages per document, unless there's some business need for the entire document. ---------------------------------------------------------------------- nartak · 2023-11-08 22:17 No kidding. If I tried to bring a product into my company without enterprise support, I'd get laughed at. It doesn't matter if the open source unsupported solution is better. The question is always: "And what happens when it breaks?" It's the difference in moving something from a development and testing environment to production. Without adequate support, the product will fail. ---------------------------------------------------------------------- skmcgowan77 · 2023-11-09 00:38 >Does paperless-ngx really support 10TB of data storage? >>Yes, but I am not sure if it was made for your sort of application This! OP, I've just started checking out paperless-ngx but have worked with SME level businesses in various consulting/contracting positions in the whole IT field. One such company was a mortgage broker in 2001,who generated a metric sh17 70n of paperwork over the course of servicing a loan. I put that out there so that you know that I have worked with this kind of thing before and I'm speaking from experience. You can't expect to run a document archiving solution that allows full text search after performing OCR on every single page you put in there and have it run very smoothly even though your generating at times 30 or more pages per document You're going to have to toss some serious hardware at that to make it run smoothly with larger docs. Also I'm pretty sure paperless was more or less designed for maybe 3 to 5 pages per document and the occasional document larger than that. Also, what kind of hardware is this running on? Is it dedicated? What's the CPU speed, number of cores, and logical cores? What's the capacity and read/write speed on your hard drives, is the data being put somewhere over the network, what speed do you actually get in that network ? what speed is the scanner, how much RAM what speed RAM,filesystem used, etc etc etc there is so much more than just plopping a piece of software on a piece of hardware of whatever design you may have laying around and expecting it to perform at tip top speed. Do ya feel me? ---------------------------------------------------------------------- skmcgowan77 · 2023-11-09 00:38 >Does paperless-ngx really support 10TB of data storage? >>Yes, but I am not sure if it was made for your sort of application This! OP, I've just started checking out paperless-ngx but have worked with SME level businesses in various consulting/contracting positions in the whole IT field. One such company was a mortgage broker in 2001,who generated a metric sh17 70n of paperwork over the course of servicing a loan. I put that out there so that you know that I have worked with this kind of thing before and I'm speaking from experience. You can't expect to run a document archiving solution that allows full text search after performing OCR on every single page you put in there and have it run very smoothly even though your generating at times 30 or more pages per document You're going to have to toss some serious hardware at that to make it run smoothly with larger docs. Also I'm pretty sure paperless was more or less designed for maybe 3 to 5 pages per document and the occasional document larger than that. Also, what kind of hardware is this running on? Is it dedicated? What's the CPU speed, number of cores, and logical cores? What's the capacity and read/write speed on your hard drives, is the data being put somewhere over the network, what speed do you actually get in that network ? what speed is the scanner, how much RAM what speed RAM,filesystem used, etc etc etc there is so much more than just plopping a piece of software on a piece of hardware of whatever design you may have laying around and expecting it to perform at tip top speed. Do ya feel me? ---------------------------------------------------------------------- modernDayKing · 2023-11-09 01:18 Hmmm first time hearing of paperless and intrigued. Looking for something to replace the soon to be extinct raven cloud. Could I use this to OCR existing pdfs as well as new scans. Think I’ll give this go this week ! ---------------------------------------------------------------------- lambadime · 2023-11-09 05:42 am configuring it using postgresql.,hoping that it can handle this huge data ---------------------------------------------------------------------- lambadime · 2023-11-09 05:43 how to limit the pages that need to be OCRed. I searched for such a settings in all possible places.,but couldnt find one in paperless. any suggestions ---------------------------------------------------------------------- padamdam · 2023-11-09 08:07 Yes it ocr everything you give it. ---------------------------------------------------------------------- MyTechAccount90210 · 2023-11-09 17:56 Yes ---------------------------------------------------------------------- zumtest99 · 2023-11-09 20:59 https://docs.paperless-ngx.com/configuration/#PAPERLESS_OCR_PAGES