ReindexException: Unable to find index micromasters-ci_public_enrollment_default

[sentry-io]

This has been going on for some time on CI.

Sentry Issue: MICROMASTERS-4V0

ReindexException: Unable to find index micromasters-ci_public_enrollment_default
  File "search/tasks.py", line 104, in index_users
    enrollments = [
  File "search/tasks.py", line 105, in <listcomp>
    enrollment for enrollment in enrollments if _document_needs_updating(enrollment)
  File "search/api.py", line 345, in document_needs_updating
    conn = get_conn()
  File "search/connection.py", line 76, in get_conn
    raise ReindexException("Unable to find index {index_name}".format(

[pdpinch]

Do you think this can be closed?

[arslanashraf7]

Do you think this can be closed?

Looked into the code as well and it was unlikely that some code change could have removed the indices that existed before so the indices might have been removed during some clean rebuild.
This was fixed by running the recreate_index command on the CI instance of Micromasters. While running the recreate_index we saw some 502s but increasing the elastic search server timeout resolved the 502s. Running the recreate_index command added the missing indices.

Yes, this can be closed, We can reopen if we see it again.

[pdpinch]

This happened again. Do we need to run recreate_index again?

[arslanashraf7]

This happened again. Do we need to run recreate_index again?

Yes, Ideally running the recreate_index should fix it and create the nonexistent indices.

[arslanashraf7]

Update, I've run the reindex again and it seems to have created the nonexistent indices, but I believe in light of this & this comments we'll have to add the recreate_index in the post deploy script in app.json & look into the Heroku build timeout OR see which app build(s) is deleting the indices because we are using the Same ES Instance for our RC/CI apps.