Argh. Well, it’s really not quite out of the box, at least on Ubutu Server 9.10 (after reinstallations required after initial failures…), so this is just a selection of the fixes that made the import and indexing of the 160 000 files at 102GB possible. So for KnowledgeTree 3.7.0.2 Commercial Edition (the same holds true for the Community Edition), the following should help:
- Use the best-practice advice when doing the local file system import – rather do 10 000 files at a time rather than 100 000 at once. Really. Trust me. It defeats the whole idea of just running a batch job. Completely. As you’d expect the option of saying – just transfer all data in directory X. But alas, that doesn’t work. So do it in batches. Manually.
- Tika Apache Indexer for Lucene – not so much on PDFs, Docs, XLS or PPT files. Install catdoc (which includes
catpptandxls2csv) andpdftotext(which you’ll find inxpdf-utils).apt-get install catdoc pdftotext- modify
knowledgetree/search2/indexing/extractors/TikaApacheExtractor.inc.phpand comment out the mime types that are affected above from the returned array ingetSupportedMimeTypes()– PDF, XLS, DOC and PPT: (more…)
