Spliting docs into paragraphs

[kamilpz]

Hi guys, I've just started playing with haystack and it's not splitting my docx files into paragraphs. Based on these lines I'm referencing looks like it's not possible. Am I missing something?

haystack/haystack/file_converter/docx.py

Lines 49 to 53 in bd823c9

	file = docx.Document(file_path) # Creating word reader object.
	paragraphs = [para.text for para in file.paragraphs]
	text = "\n".join(paragraphs)
	document = {"content": text, "content_type": "text", "meta": meta}
	return document

haystack/haystack/preprocessor/utils.py

Lines 294 to 300 in bd823c9

	if split_paragraphs:
	for para in text.split("\n\n"):
	if not para.strip(): # skip empty paragraphs
	continue
	documents.append({"content": para, "meta": {"name": path.name}})
	else:
	documents.append({"content": text, "meta": {"name": path.name}})

[julian-risch]

Hi @kamilpz I can recommend two resources to read as starting points regarding splitting of documents.

1. Our blog article on Parameter Tweaking has a section "Increasing Pipeline Speed via Document Length Optimization", which might be interesting to you: https://www.deepset.ai/blog/parameter-tweaking-get-faster-answers-from-your-haystack-pipeline
2. We have a tutorial on preprocessing, where the split_by="word" parameter is set but you can easily change it to split_by="passage": https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial8_Preprocessing.ipynb

While the file converter converts your docx file into text in string format, the preprocessor handles splitting long text inputs into smaller chunks that can then be processed by the neural network models. The splitting can take into account word, sentence or paragraph (passage) boundaries. Within haystack, we call these smaller chunks "documents" and store them in document stores. So your original docx file will most likely be stored as multiple documents within haystack.

I hope this answers your questions and if not feel free to describe in more detail what you would like to achieve/ what's your use case and I'd be happy to help.

[kamilpz]

Thanks @julian-risch for this rich answer. My point was that when docx converter joins paragraphs by "\n" then convert_files_to_dicts method of utils is not able to split them back because it expects two new lines "\n\n". But I started using PreProcessor class just as you suggested and I'm splitting not on paragraphs level but on sentences level which is good idea and works just fine.