Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Whoosh 2.7 x python 3.6 refactoring #537

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

Whoosh 2.7 x python 3.6 refactoring #537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
581ebc2
remove whoosh 0.3
assem-ch Apr 19, 2021
4e32c33
refactoring python 3 x whoosh 2.7
assem-ch Sep 4, 2021
2ac3ecc
remove unused exception
assem-ch Sep 4, 2021
6c9205e
remove loggin errors
assem-ch Sep 4, 2021
c405cd1
fix numeric values
assem-ch Sep 5, 2021
5375c95
convert dynamic resources to be json instead of python and clean logi…
assem-ch Sep 6, 2021
d9610a9
fix highlight and adjacent ayas
assem-ch Sep 10, 2021
0afaecd
resources as json
assem-ch Nov 12, 2021
7be73de
rename araby
assem-ch Nov 30, 2021
0137679
fixes for importer
assem-ch Nov 30, 2021
fa90e93
fixes
assem-ch Dec 4, 2021
d709dd4
fix suggestion
assem-ch Dec 4, 2021
f21de87
fix autocomplete
assem-ch Dec 4, 2021
e5d8025
accept not same prefix for suggestion
assem-ch Dec 4, 2021
6c6b4a1
Merge pull request #540 from Alfanous-team/res_as_json
assem-ch Dec 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .checkignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
src/alfanous-import/initial_importing.py
src/alfanous/Support/*

2 changes: 1 addition & 1 deletion .github/dependabot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# To get started with Dependabot version updates, you'll need to specify which
# To get started with Dependabot version updates, yo'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ dynamic_resources/

# generated indexes #
#####################
*.seg
*_WRITELOCK
_*.dcz
_*.dci
_*.pst
Expand Down Expand Up @@ -120,6 +122,7 @@ information.json




# Logs and databases #
######################
*.log
Expand All @@ -137,4 +140,5 @@ Icon?
*.db
desktop.ini
venv/
venv2/

120 changes: 13 additions & 107 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
# This is Alfanous Makefile
#


## Global Version of the project, must be updated in each significant change in
## the API & Desktop Gui
VERSION=0.7.30
Expand All @@ -15,19 +14,12 @@ API_PATH="./src/"

## Importer package path, the importer is the responsible of downloading quranic
## resources, updating them , indexing them.
QIMPORT=$(API_PATH)"alfanous-import/cli.py"
QIMPORT=$(API_PATH)"alfanous_import/cli.py"

## Dynamic resources, are some auto-generated python modules that contain some
## linguistic resources on the form of python dictionaries
DYNAMIC_RESOURCES_PATH=$(API_PATH)"alfanous/dynamic_resources/"

## resources & configuration files
## resources & configuration files
CONFIGS_PATH=$(API_PATH)"alfanous/configs/"
RESOURCES_PATH=$(API_PATH)"alfanous/resources/"

## Databases file, the databases mainly store the Quranic data in a middle
## structure before indexing it
DB_PATH="./resources/databases/"

## Store is the place where we're storing the Quranic data in its original
## structure (raw data)
Expand All @@ -38,19 +30,14 @@ INDEX_PATH=$(API_PATH)"alfanous/indexes/"



## Python version and command
PYTHON_VERSION_MAJ=$(shell python -c 'import sys; print(sys.version_info[0])')
ifeq '$(PYTHON_VERSION_MAJ)' '2'
PYTHON_COMMAND="python"
else
PYTHON_COMMAND="python2"
endif


## default target, it's what to do if you typed "make" without target
default:
@echo "choose a target:"
@echo "edit_all \n\t to edit all resources that should humanly edited "
@echo "download_all \n\t to download all Quranic resources that we can't \n\t include in Git or tarball because of license or huge size"
@echo "edit \n\t to edit all resources that should humanly edited "
@echo "download \n\t to download all Quranic resources that we can't \n\t include in Git or tarball because of license or huge size"
@echo "build \n\t to build all indexes, update all resources, qt files, \n\t localization files"
@echo "dist \n\t to generate all distribution files for the API and \n\t the Desktop interface"

Expand All @@ -59,7 +46,7 @@ default:
# 1. api information, see update_information
# 2. globl hints (deprecated), see update_hints
# 3. List of translations to be downloaded, see update_translations_to_download_list
edit_all: edit_information edit_hints edit_translations_to_download_list
edit: edit_information edit_hints edit_translations_to_download_list

# update information manually
edit_information:
Expand All @@ -79,9 +66,8 @@ edit_translations_to_download_list:
## this target is to build all what have to be built:
# 1. Update Quranic resources needed for indexing phase, see update_pre_build
# 2. Generate all Indexes, see index_all
# 3. Generate all spelling dictionaries, see speller_all
# 4. Update all resources that must be updated after indexing phase or independently, see update_post_build
build: update_pre_build index_all speller_all update_post_build
build: update_pre_build index_all update_post_build



Expand All @@ -103,7 +89,7 @@ clean_all: clean_deb
## download Quranic resources needed for Alfanous project, which are:
# 1. Quran translations from zekr.org, see download_translations
# 2.
download_all: download_translations download_tanzil
download: download_translations download_tanzil

download_translations:
# download from http://zekr.org/resources.html to ./store/traductions
Expand All @@ -118,20 +104,14 @@ download_tanzil:

## update resources that must be updated after (or independent to) indexing phase, which are:
# 1. list of indexed translations, see update_translations_indexed_list
# 2. list of offline recitations, see update_recitations_offline_list
# 3. list of online recitations, see update_recitations_online_list
update_post_build: update_dynamic_resources_postbuild update_recitations_offline_list #update_recitations_online_list
# 3. list of online recitations, see update_recitations_online_list
update_post_build: update_dynamic_resources_postbuild #update_recitations_online_list

## update resources that must be updated before indexing phase, which are:
# 1. Quranic Arabic Corpus, see update_quranic_corpus
# 2. Linguistic resources on the form of python dictionarries to accelerate the loading , see update_dynamic_resources

update_pre_build: construct_database update_translations_indexed_list update_dynamic_resources_prebuild #update_quranic_corpus

# Construct database from dump file
construct_database:
cd $(DB_PATH); rm main.db || true ; cat main.sql | sqlite3 main.db
perl -p -w -e 's|alfanous.release|$(RELEASE)|g;s|alfanous.version|$(VERSION)|g;' $(API_PATH)alfanous/resources/information.json.in > $(API_PATH)alfanous/resources/information.json
update_pre_build: update_translations_indexed_list

# update list of indexed translations automatically using Importer
update_translations_indexed_list:
Expand All @@ -142,60 +122,12 @@ update_translations_indexed_list:
update_quranic_corpus:
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -u wordqc $(STORE_PATH)quranic-corpus-morpology.xml $(DB_PATH)main.db

# update recitations offline list TODO
update_recitations_offline_list:
@echo "todo"


# update recitations online list automatically from the project every ayah
update_recitations_online_list:
cd $(CONFIGS_PATH); wget http://www.everyayah.com/data/recitations.js; mv recitations.js recitations.json

# update dynamic resources automatically , see transfer_prebuild, transfer_postbuild
update_dynamic_resources_prebuild: transfer_prebuild
update_dynamic_resources_postbuild: transfer_postbuild

## build dynamic_resources by transferring the data from database (or indexes) to python modules, this include:
# 1. [prebuild] arabic stop words, see transfer_stopwords
# 2. [prebuild] Quranic words synonyms, see transfer_synonyms
# 3. [prebuild] Word properties [root,type], see transfer_word_props
# 4. [prebuild] Derivtion levels of Qurnic words, see transfer_derivations
# 5. [prebuild] Field names mapping Arabic to English, see transfer_ara2eng_names
# 6. [prebuild] Quranic words mapping Standard to Uthmani, see transfer_standard2uthmani
# 7. [postbuild] Different vocalizations of each quranic word, see transfer_vocalizations
transfer_all: transfer_prebuild transfer_postbuild
transfer_prebuild: transfer_stopwords transfer_synonyms transfer_word_props transfer_derivations transfer_ara2eng_names transfer_standard2uthmani
touch $(DYNAMIC_RESOURCES_PATH)__init__.py
transfer_postbuild: transfer_vocalizations

transfer_stopwords:
mkdir -p $(DYNAMIC_RESOURCES_PATH); touch $(DYNAMIC_RESOURCES_PATH)__init__.py
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -t stopwords $(DB_PATH)main.db $(DYNAMIC_RESOURCES_PATH)

transfer_synonyms:
mkdir -p $(DYNAMIC_RESOURCES_PATH);touch $(DYNAMIC_RESOURCES_PATH)__init__.py
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -t synonyms $(DB_PATH)main.db $(DYNAMIC_RESOURCES_PATH)

transfer_word_props:
mkdir -p $(DYNAMIC_RESOURCES_PATH);touch $(DYNAMIC_RESOURCES_PATH)__init__.py
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -t word_props $(DB_PATH)main.db $(DYNAMIC_RESOURCES_PATH)

transfer_derivations:
mkdir -p $(DYNAMIC_RESOURCES_PATH);touch $(DYNAMIC_RESOURCES_PATH)__init__.py
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -t derivations $(DB_PATH)main.db $(DYNAMIC_RESOURCES_PATH)

transfer_vocalizations:
mkdir -p $(DYNAMIC_RESOURCES_PATH);touch $(DYNAMIC_RESOURCES_PATH)__init__.py
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -t vocalizations $(DB_PATH)main.db $(INDEX_PATH)main/ $(DYNAMIC_RESOURCES_PATH)

transfer_ara2eng_names:
mkdir -p $(DYNAMIC_RESOURCES_PATH);touch $(DYNAMIC_RESOURCES_PATH)__init__.py
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -t ara2eng_names $(DB_PATH)main.db $(DYNAMIC_RESOURCES_PATH)

transfer_standard2uthmani:
mkdir -p $(DYNAMIC_RESOURCES_PATH);touch $(DYNAMIC_RESOURCES_PATH)__init__.py
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -t std2uth_words $(DB_PATH)main.db $(DYNAMIC_RESOURCES_PATH)



## build all indexes:
# 1. Main index that contains all information related to Ayah or Surah, see index_main
Expand All @@ -205,35 +137,10 @@ index_all: index_main index_extend #index_word
@echo "done;"

index_main:
export PYTHONPATH=$(API_PATH) ; rm -r $(INDEX_PATH)main/; $(PYTHON_COMMAND) $(QIMPORT) -x main $(DB_PATH)main.db $(INDEX_PATH)main/
chmod 644 $(INDEX_PATH)main/*_LOCK
export PYTHONPATH=$(API_PATH) ; rm -r $(INDEX_PATH)main/; $(PYTHON_COMMAND) $(QIMPORT) -x main $(RESOURCES_PATH) $(INDEX_PATH)main/

index_extend:
export PYTHONPATH=$(API_PATH) ; rm -r $(INDEX_PATH)extend/; $(PYTHON_COMMAND) $(QIMPORT) -x extend $(STORE_PATH)translations/ $(INDEX_PATH)extend/
chmod 644 $(INDEX_PATH)extend/*_LOCK

index_word:
export PYTHONPATH=$(API_PATH) ; rm -r $(INDEX_PATH)word/; $(PYTHON_COMMAND) $(QIMPORT) -x word $(DB_PATH)main.db $(INDEX_PATH)word/
chmod 644 $(INDEX_PATH)word/*_LOCK

## build all spellers:
# 1. Speller of ayah unvocalized standard text words, see speller_aya
# 2. Speller of subject fields (deprecated), see speller_subject
# 3. Speller of quranic unvocalized uthmani words, see speller_word
speller_all: speller_aya speller_subject #speller_word
@echo "done;"

speller_aya:
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -p aya $(INDEX_PATH)main/
chmod 644 $(INDEX_PATH)main/*_LOCK

speller_subject:
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -p subject $(INDEX_PATH)main/
chmod 644 $(INDEX_PATH)main/*_LOCK

speller_word:
export PYTHONPATH=$(API_PATH) ; $(PYTHON_COMMAND) $(QIMPORT) -p word $(INDEX_PATH)word/
chmod 644 $(INDEX_PATH)word/*_LOCK


install:
Expand All @@ -248,6 +155,5 @@ dist:
@echo "NOTE: you can find the generated egg in ./output"



release: dist

34 changes: 0 additions & 34 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,40 +458,6 @@ Translations
TODO


Information
^^^^^^^^^^^
flags:

.. code-block:: python

{
"action"="show",
"query"="information"
}

response:

.. code-block:: python

{
"show": {
"information": {
"engine": "Alfanous",
"wiki": "http://wiki.alfanous.org/doku.php?id=json_web_service",
"description": "Alfanous is a Quranic search engine provides simple and advanced search services in the diverse information of the Holy Quran .",
"author": "Assem chelli",
"version": "0.5",
"contact": "[email protected]",
"console_note": "this is console interface of Alfanous, try -h to get help ",
"json_output_system_note": "\n This is the <a href='http://json.org/'>JSON</a> output system of <a href=\"http://wiki.alfanous.org\">Alfanous</a> project .This feature is in Alpha test and the Json schema may be it's not stable . We are waiting for real feadbacks and suggestions to improve its efficacity,quality and stability. To contact the author ,please send a direct email to <b> assem.ch[at]gmail.com</b> or to the mailing list <b>alfanous [at] googlegroups.com</b>\n <br/><br/> For more details visit the page of this service <a href=\"http://wiki.alfanous.org/doku.php?id=json_web_service\">here</a>\n "
}
},
"error": {
"msg": "success ## action=show ; query=information",
"code": 0
}
}


Recitations
^^^^^^^^^^^
Expand Down
Loading