Skip to content

Commit

Permalink
fix(elasticsearch-ingestion): detect sub-properties in 'nested' type …
Browse files Browse the repository at this point in the history
…mapping

fix issue where sub-properties in 'nested' type mappings are not detected during
Elasticsearch ingestion, resulting in incomplete ingestion of data. Prior to this
fix, only top-level properties were recognized, missing critical sub-fields.

Fixes: datahub-project#10911
  • Loading branch information
Lawrence De Spiegeleire authored and Bumyu committed Sep 13, 2024
1 parent 54b8ce3 commit e1bd272
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ def _get_schema_fields(
for columnName, column in elastic_schema_dict.items():
elastic_type: Optional[str] = column.get("type")
nested_props: Optional[Dict[str, Any]] = column.get(PROPERTIES)
if elastic_type is not None:
if elastic_type is not None and elastic_type != "nested":
self._prefix_name_stack.append(f"[type={elastic_type}].{columnName}")
schema_field_data_type = self.get_column_type(elastic_type)
schema_field = SchemaField(
Expand Down

0 comments on commit e1bd272

Please sign in to comment.