[HUDI-6891] Fix RO queries with RLI and record key predicate

[codope]

Change Logs

Suppose RLI (record index) is enabled on MOR table, do an insert and then delete a record. When table is not compacted, it is expected that RO (read-optimized) query shows the deleted record, while snapshot query will not. A query with no predicates works fine, but if the query has a predicate on record key field, it will use RLI and RLI always returns the latest snapshot. Hence, this PR fixes the issue by not using RLI for queries other than snapshot queries. The PR adds a new supportsQueryType(queryType: HoodieTableQueryType) API in the base class for index support. Different index can implement it differently as and when we add time tavel support for indexes.

Impact

Fix RO queries with RLI and record key predicate.

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

A query with no predicates works fine, but if the query has a predicate on record key field, it will use RLI and RLI always returns the latest snapshot. Hence, this PR fixes the issue by not using RLI

Can we just skip the file filtering if we find the file in the index item is a log file when rdo RO query, so that we can eliminate the new API on the file index. And a RO query on full compacted table can also utilitize the RLI index.

[codope]

Can we just skip the file filtering if we find the file in the index item is a log file when rdo RO query, so that we can eliminate the new API on the file index. And a RO query on full compacted table can also utilitize the RLI index.

@danny0405 This is a good point. But, the index item in RLI only contains fileId and not the file name. Moreover, I still think it makes sense to have this new API because think about time travel query. Let's say there is an instant t in the past which is a compaction instant and table was fully compacted at that instant. If user runs a query with record key predicate and as of instant t then RLI would still return candidate files as per the latest snapshot. And RO queries are nothing but time travel as of the latest compaction time.

I think it's much cleaner to simply not use RLI for any type of queries other than Snapshot queries. Later, when we add support for time travel on metadata table, we can easily change the condition in the implementation of this API and use the index.

[hudi-bot]

CI report:

• c16f9ec Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

Then can we just handle this logic in RLI index because the issue is specific to it? Just skip the data skipping inside RLI index should be working right? We can introduce the new API if more index type are involved I think.

[codope]

Then can we just handle this logic in RLI index because the issue is specific to it? Just skip the data skipping inside RLI index should be working right? We can introduce the new API if more index type are involved I think.

It's for RLI and Secondary index. I can remove the colstats override. The default implementation of API returns true which should be fine.