Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(sql): insert now works with specific database/catalog #9821

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

viamiraia
Copy link

self.get_schema should be passed the database and catalog. When doing an insert into a duckdb table specific to a database, this line causes a table not found error.

Description of changes

One line was modified to pass database and catalog to the _build_insert_from_table function in the SQL backend.

self.get_schema should be passed the database and catalog. When doing an insert into a duckdb table specific to a database, this line causes an error.
@cpcloud
Copy link
Member

cpcloud commented Aug 12, 2024

👋🏻 Thanks for the PR!

Would you mind adding a test for this that fails without your change and then passes with it?

@viamiraia
Copy link
Author

Sure, I'm currently low on time; would writing it here be ok?

import ibis
import polars as pl

con = ibis.duckdb.connect('test.duckdb')
con.create_database('test_db')

test_schema = ibis.schema({'a': 'int64', 'b': 'string'})
con.create_table('test_table', database='test_db', schema=test_schema)

df_to_insert = pl.DataFrame({
    'a': [0,1], 
    'b': ['z','y'],
    }, schema=test_schema.to_polars())

con.insert('test_table', df_to_insert, database='test_db')

In 9.3.0 this errors out with:

---------------------------------------------------------------------------
CatalogException                          Traceback (most recent call last)
File c:\test\.pixi\envs\dev\Lib\site-packages\ibis\backends\duckdb\__init__.py:318, in Backend.get_schema(self, table_name, catalog, database)
    317 try:
--> 318     result = self.con.sql(query)
    319 except duckdb.CatalogException:

CatalogException: Catalog Error: Table with name test_table does not exist!
Did you mean "test_db.test_table"?

During handling of the above exception, another exception occurred:

IbisError                                 Traceback (most recent call last)
Cell In[16], line 2
      1 df_to_insert = pl.DataFrame({'a': [0,1], 'b': ['z','y']}, schema=test_schema.to_polars())
----> 2 con.insert('test_table', df_to_insert, database='test_db')

File c:\test\.pixi\envs\dev\Lib\site-packages\ibis\backends\sql\__init__.py:421, in SQLBackend.insert(self, table_name, obj, schema, database, overwrite)
    417     obj = ibis.memtable(obj)
    419 self._run_pre_execute_hooks(obj)
--> 421 query = self._build_insert_from_table(
    422     target=table_name, source=obj, db=db, catalog=catalog
    423 )
    425 with self._safe_raw_sql(query):
    426     pass

File c:\test\.pixi\envs\dev\Lib\site-packages\ibis\backends\sql\__init__.py:439, in SQLBackend._build_insert_from_table(self, target, source, db, catalog)
    433 # Compare the columns between the target table and the object to be inserted
    434 # If they don't match, assume auto-generated column names and use positional
    435 # ordering.
    436 source_cols = source.columns
    437 columns = (
    438     source_cols
--> 439     if not set(target_cols := self.get_schema(target).names).difference(
    440         source_cols
    441     )
    442     else target_cols
    443 )
    445 query = sge.insert(
    446     expression=self.compile(source),
    447     into=sg.table(target, db=db, catalog=catalog, quoted=quoted),
    448     columns=[sg.to_identifier(col, quoted=quoted) for col in columns],
    449     dialect=compiler.dialect,
    450 )
    451 return query

File c:\test\.pixi\envs\dev\Lib\site-packages\ibis\backends\duckdb\__init__.py:320, in Backend.get_schema(self, table_name, catalog, database)
    318     result = self.con.sql(query)
    319 except duckdb.CatalogException:
--> 320     raise exc.IbisError(f"Table not found: {table_name!r}")
    321 else:
    322     meta = result.fetch_arrow_table()

IbisError: Table not found: 'test_table'

@cpcloud cpcloud self-assigned this Aug 28, 2024
@cpcloud cpcloud marked this pull request as draft September 5, 2024 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants