Page History

...

Tuning the Data Warehouse

pgtune (optional)

Although specifying exact postgresql.conf configuration parameters may seem beneficial, in reality, every installation is different. There are many factors in tuning your cluster including system memory, resource contention from other services running on the server, available disk space, disk types et al. The primary cluster supporting your updates was tuned using the tool pgtune. It is recommended to run the tool and apply the changes before bulk loading the data. Note that this requires Python.

yum install pgtune
  pgtune -i postgresql.conf  -T DW -c 500
  [ paste added configuration options at end of postgres.conf ]

Redirecting errors to LOG.2 (optional)

If desired, you can redirect errors (if any) to LOG.2:

cat alexandria-dwh.sql | psql -Ualexandria postgres 2>LOG.2

Pre-flight check

Use the pre-flight check to ensure the database and operating system are properly configured before loading the tables:

./pre-flight-check.sh

The sample output of a properly configured system would look like this:

# Testing localhost/alexandria ...
#   OK    : procedual language sql
#   OK    : procedual language plpgsql
#   OK    : procedual language plperl
#   OK    : procedual language plperlu
#   OK    : XML capability (test 1/libxml): 
#   OK    : XML capability (test 2/libxml): 

Loading the Tables

The process of loading CLAIMS Direct data into PostgreSQL tables will take 31-5 2 days on most hardware configurations.

...

# for each table, load ...
gunzip -c data/<table>.gz | psql -Ualexandria alexandria
 
# sample batch script
for tbldata in $(ls data/*.gz); do
  gunzip -c $tbldata | psql -Ualexandria alexandria
done