Native TEXTFILE reader produces incorrect results when varchars contain the field separator escape character

[alexjo2144]

Here's a test case:

    @Test
    public void testTextfileFieldSeparator()
    {
        List<String> specialCharacterValues = ImmutableList.of(
                "1, 'a,comma'",
                "2, 'a|pipe'",
                "3, 'an''escaped quote'",
                "4, 'a~null encoding'");

        Session withHadoopReaders = Session.builder(getSession())
                .setCatalogSessionProperty("hive", "json_native_reader_enabled", "false")
                .setCatalogSessionProperty("hive", "text_file_native_reader_enabled", "false")
                .build();
        Session withNativeReaders = Session.builder(getSession())
                .setCatalogSessionProperty("hive", "json_native_reader_enabled", "true")
                .setCatalogSessionProperty("hive", "text_file_native_reader_enabled", "true")
                .build();

        try (TestTable table = new TestTable(
                getQueryRunner()::execute,
                "test_textfile_field_separator",
                "(id INT, varchar_t VARCHAR) WITH (format = 'TEXTFILE', textfile_field_separator=',', textfile_field_separator_escape='|', null_format='~')",
                specialCharacterValues)) {
            assertQuery(withHadoopReaders, "SELECT id FROM " + table.getName() + " WHERE varchar_t = 'a,comma'", "VALUES 1");
            assertQuery(withHadoopReaders, "SELECT id FROM " + table.getName() + " WHERE varchar_t = 'a|pipe'", "VALUES 2");
            assertQuery(withHadoopReaders, "SELECT id FROM " + table.getName() + " WHERE varchar_t = 'an''escaped quote'", "VALUES 3");
            assertQuery(withHadoopReaders, "SELECT id FROM " + table.getName() + " WHERE varchar_t = 'a~null encoding'", "VALUES 4");

            assertQuery(withNativeReaders, "SELECT id FROM " + table.getName() + " WHERE varchar_t = 'a,comma'", "VALUES 1");
            // This assertion fails
            assertQuery(withNativeReaders, "SELECT id FROM " + table.getName() + " WHERE varchar_t = 'a|pipe'", "VALUES 2");
            assertQuery(withNativeReaders, "SELECT id FROM " + table.getName() + " WHERE varchar_t = 'an''escaped quote'", "VALUES 3");
            assertQuery(withNativeReaders, "SELECT id FROM " + table.getName() + " WHERE varchar_t = 'a~null encoding'", "VALUES 4");
        }
    }

[alexjo2144]

FYI @dain

[kokosing]

Could #18254 unblock the release?