You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

The PostgreSQL component is the heart CLAIMS Direct. It contains the XML for the entire data warehouse collection, processes updates from the primary and functions as data source for the optional SOLR index.

Hardware Requirements

Requirement
Recommended
CPU4-cores
System Memory24GB
Storage Capacity4TB (SSD prefered)

 Software Requirements

Requirement
Minimum Version
Notes
Operating SystemRHEL 6, Fedora 20, Centos 6 
Development ToolsDistribution version
yum|dnf groupinstall "Development Tools"
PostgreSQLDistribution version
 yum|dnf install \
    postgresql postgresql-contrib \
    postgresql-odbc postgresql-pl-perl \
    postgresql-server
System LibrariesDistribution version
yum|dnf install \
libxml2 libxml2-devel \
libxslt libxslt-devel
# Please see note below regarding libxml2
Perl and ModulesDistribution version
yum|dnf install \
perl-Module-Install \ perl-DBD-Pg \ perl-XML-LibXML \ perl-XML-LibXSLT \ perl-CPAN

libxml2

Some CLAIMS Direct loading and maintenance code utilizes the postgresql perl extension (plperl) as well as a heavy reliance on the libxml2 XML parsing library. The following table lists some inconsistent behavior with disparate versions of postgresql and libxml2.

8.4.62.7.8works
9.1.22.7.8works
9.1.42.7.8works
9.1.72.7.8fails
9.2.42.9.1yes
9.3.12.7.6fails
9.3.22.9.1works

No postgresql version compiled with libxml2 < 2.7.8 works and additionally, postgresql 9.1.7 fails even with libxml2 2.7.8

IFI Claims has produced a patched release of libxml2-2.9.2 as an RPM. It is recommended to locally install this package replacing the package in the distribution. The RPM can be downloaded at the URL: http://alexandria.fairviewresearch.com/software/libxml2/f20/libxml2-2.9.2-1.fc20.x86_64.rpm. For additional versions, please contact support@ificlaims.com.

Postgresql Installation

Regardless of installation type, careful planning of disk resources is important for efficient data loading into and extraction out of postgresql. There are 6 logical segments inside the CLAIMS Direct data warehouse.

work indexAll indices pertaining to loading.30GB (variable)
work textAll raw table data queued for loading.100GB (variable)
xml indexAll permanent indices for the data warehouse400GB
xml textAll permanent text for the data warehouse1TB
pg dataThe cluster meta data, reporting and logging directory5GB (variable)
pg xlogLog shipping for replication50GB (variable)

Each of these segments can be allocated discrete disk through the use of TABLESPACES. Although not required, the use of TABLESPACES will improve loading and extraction performance. The total logical size of the data warehouse is approximately 2TB after initial loading.

Suggested Postgresql Disk Layout

As mentioned above, the CLAIMS Direct postgresql cluster can utilize TABLESPACES to separate text, index and work table data. An optimal (but not mandatory) layout will have each of the following paths on separate disk groups where "disk group" is understood to be a discreet disk or set of disks exposed to the operating system as a device capable of supporting an ext4 file system.

Please note, these are only suggestions. Your environment and disk sub-system naming may be different or, you can choose not to use TABLESPACES at all. A postgresql cluster running on a 2TB RAID0 sub-system exposed as one device, for example, wouldn't benefit as noticeably using TABLESPACES as a mixed RAID environment with multiple devices.

If you chose to implement TABLESPACEs, you will need to edit alexandria-dwh.sql and insert your custom tablespace definitions. These are clearly marked per segment as:

 

-- TABLESPACE definition for work text
-- TABLESPACE definition for xml text
-- TABLESPACE definition for work index
-- TABLESPACE definition for xml index
 
-- insert your TABLESPACE name in the appropriate section
set default_tablespace = ''
tblspc_wrkidxwork index/wrkidx/_pg30GB
tblspc_wrktxtwork text/wrktxt/_pg100GB
tblspc_xmlidxxml index/xmlidx/_pg400GB
tblspc_xmltxtxml text/xmltxt/_pg2000GB

During cluster initialization (initdb), you can further define the location for the segments: pg data and pg xlog.

Postgresql Initial Configuration

CLAIMS Direct requires a working postgresql cluster. If you do not have an initialized cluster, the following steps will initialize the cluster and give you rudimentary authentication and access levels needed to run CLAIMS Direct.

 

su - postgres
 
# -D, --pgdata: specify location of pg data segment
# -X, --xlogdir: sepcify location of pg xlog segment
initdb -A trust \
       -D /var/lib/pgsql/data \
       -E utf8 \
       -X /pgxlog

 

 

The following configuration files are to be edited

/var/lib/pgsql/data/pg_hba.conf

 

# Allow trusted local access
local   all             all                                     trust
 
# IPv4 local connections
host    all             all             127.0.0.1/32            trust
 
# Other hosts on subnets that may require access, for example
# host    all             all             192.168.10.0/24         trust

 

 

/var/lib/pgsql/data/postgresql.conf

 

listen_addresses = '*'                  # what IP address(es) to listen on;

 

 

If you already have an initialized cluster, please be certain that local access is enabled for stand-alone installation. In either distributed install, if a separate services machine is created, its IP address needs access. This is imperative for the client update procedures.

 

Finally, enable and start the postgresql cluster

 

# on systemd based systems:
systemctl enable postgresql.service
 
systemctl start postgresql.service

 

 

CLAIMS Direct postgresql role

CLAIMS Direct requires the role alexandria

 

psql -Upostgres postgres
=#> create role alexandria with superuser login;

 

Creating the Database

The postgresql data warehouse portion of CLAIMS Direct is delivered in 2 parts:

  •  postgresql database schema (alexandria-dwh.sql)
  • <table>.gz files located in the sub-directory data

 

# create the database
cat alexandria-dwh.sql | psql -Ualexandria postgres

 

Tuning the Data Warehouse

Although specifying exact postgresql.conf configuration parameters may seem beneficial, in reality, every installation is different. There are many factors in tuning your cluster including system memory, resource contention from other services running on the server, available disk space, disk types et al. The primary cluster supporting your updates was tuned using the tool pgtune. It is recommended to run the tool and apply the changes before bulk loading the data.

Load the Tables

The process of loading CLAIMS Direct data into postgresql tables will take 3-5 days on most hardware configurations.

 
 The following code block outlines the loading of the CLAIMS Direct data into postgresql tables.

 

# for each table, load ...
gunzip -c data/<table>.gz | psql -Ualexandria alexandria
 
# sample batch script
for tbldata in $(ls data/*.gz); do
  gunzip -c $tbldata | psql -Ualexandria alexandria
done

Creating the Credentials Database for CDWS

TODO: sql package, installation and population instructions.

 

  • No labels