Skip to content

Latest commit

 

History

History
31 lines (24 loc) · 1.65 KB

README.md

File metadata and controls

31 lines (24 loc) · 1.65 KB
Raw

Bulk Data Import

This is a small demo to illustrate how to load data in bulk into a PostgreSQL database. It was inspired by a live demo to import data that at first led to this Gist.

Steps to reproduce are:

  1. Create the demo schema
  2. Create (random) sample data. We will end up with csv-Files that match our schema.
  3. Import data into PostgreSQL

Schema Lectures

model

Load Data into PostgreSQL

As always, there are multiple options to import data as bulk.

  1. Use a tool to import csv-files directly (e.g., pgAdmin can do that)
  2. Use psql to do it; the \copy command is your friend.
  3. Use a query to do do; similarly to psql, you may use the COPY statement.
  4. Use a tool to convert csv into INSERT statements (e.g., tableconvert can do that)
  5. Use any programming language (e.g., Java or Python) and write a short program snippet to do it for you.

File descriptions

File Description
create_data.py Simple Python script to generate random data
create_environment.sh Bash script to create a Python environment and execute the data generation
create_schema.sql Schema to create a database matching the generated data
load_data.py Simple Python script to load data into a database. Keep in mind that you need to configure your connection.
load_data.sql Command to load data from psql
model.dbm ER model that depicts the schema (created with pgmodeler)