AMA Megathread: Getting Started with dbt in 2022

[krevitt]

New to dbt? A member of the dbt Labs team will be in this thread every Thursday @ 1pm ET (starting 2022-09-08) to answer your burning questions.

Update 2022-11-01:
This thread has now moved to dbt Community Slack (in the #dbt-101-session channel) - see y'all there!

[Victoriapm]

Hello EMEA, I'll be monitoring this thread today and make sure no question goes unanswered ☎️ 🍿

[hdotking]

Hello @Victoriapm

I’ve been using DBT Core for the past few months and have been loving the experience 👍

I’m seeing tonnes of work going into DBT cloud nowadays and the UI looks super useful for me and my team. However the main thing stopping me from using DBT cloud is the difficulty in setting it up from scratch…
I must be doing something wrong because whenever I try to Test Connection under the Configure Your Environment heading I get a timeout error:

Runtime Error
  dbt was unable to connect to the specified database.
  The database returned the following error:
  
    >Database Error
    timeout expired
    
  
  Check your database credentials and try again. For more information, visit:
  https://docs.getdbt.com/docs/configure-your-profile

Do you know what could be causing this error and how I can get past it?

Thank you!

[Victoriapm]

Hi @hdotking and thanks for posting the first question 🎉 I'm so excited that you're loving the experience and appreciate the work towards dbt cloud!!
I have an idea of what could be happening, but would you please tell me first which database are you trying to connect to? Is this Redshift? Also, can you see any attempt to log in in your database query monitor?

[Victoriapm]

Hey @hdotking just wanted to follow up here, and recommend go directly to our customer support who will happily help you to get dbt cloud up and running with your database soon!

[nikotime]

Hey, got a beginner question about threads and cost! We have just set up DBT in Fivetran with transformations going to an XS database in Snowflake. I note that the max threads that can be used for this size is 8, are there any cost implications for going for different numbers of threads between 1 and 8 for our transformations? I just left it on 1 to start with as we have very few transformations thus far!

[Victoriapm]

Hey Niko, that's a very good question. The thing with dbt cloud threads is stating how many concurrent queries will dbt send to your data warehouse but it will be highly affected by the capacity of your data warehouse and the actual models in the dbt run selection.
Let me exemplify to make it more clear:

1. It will depend on how many models can run at the same time due to dependencies: If I do a dbt run --select model_a+ where the lineage looks very linear and straight forward, one model after another, something like model_a->model_b->model_c. Regardless of the threads, it will run only one model at the time.
2. It will be restricted by your data warehouse capacity: If I do a dbt run where 8 models exist in that selection that can run concurrently, dbt will send those 8 queries, but if your warehouse size is XS, in reality it will run 2 models at the time and the rest will queue on the snowflake query monitor.

Now that that's clear (hopefully), when running those dbt transformations in a XS warehouse, dbt should not incur extra cost in the Snowflake side as you increase the threads of your run. I do recommend to have a look at the scalability options of your warehouse though, to make sure that because of that thread increase you're not causing a resize unexpectedly as so many more queries will queue.

[nikotime]

Hey, thanks so much for your detailed answer! I'm running a Hubspot package maintained by Fivetran which looks quite complex, so have increased to 6 threads given it wouldn't incur extra cost (being very careful not to push Snowflake to upgrade to an S!).

[Victoriapm]

As I'm now logging off, I wanted to share that we'll keep receiving the alerts if you raise more questions and answering back, but not as promptly as I won't be actively checking the thread.
Thanks for the questions, happy dbt'ing and I'll leave with a small tip in form a meme (from the #memes-and-off-topic-chatter ofc)