Improving Performance at mapping level

Optimizing Datatype Conversions

Forcing the Informatica Server to make unnecessary datatype conversions slows performance.

For example, if your mapping moves data from an Integer column to a Decimal column, then back to an Integer column, the unnecessary datatype conversion slows performance. Where possible, eliminate unnecessary datatype conversions from mappings.

Some datatype conversions can improve system performance. Use integer values in place of other datatypes when performing comparisons using Lookup and Filter transformations.

For example, many databases store U.S. zip code information as a Char or Varchar datatype. If you convert your zip code data to an Integer datatype, the lookup database stores the zip code 94303-1234 as 943031234. This helps increase the speed of the lookup comparisons based on zip code.


Optimizing Lookup Transformations
If a mapping contains a Lookup transformation, you can optimize the lookup. Some of the things you can do to increase performance include caching the lookup table, optimizing the lookup condition, or indexing the lookup table.

Caching Lookups
If a mapping contains Lookup transformations, you might want to enable lookup caching. In general, you want to cache lookup tables that need less than 300MB.
When you enable caching, the Informatica Server caches the lookup table and queries the lookup cache during the session. When this option is not enabled, the Informatica Server queries the lookup table on a row-by-row basis. You can increase performance using a shared or persistent cache:

Shared cache. You can share the lookup cache between multiple transformations. You can share an unnamed cache between transformations in the same mapping. You can share a named cache between transformations in the same or different mappings.
Persistent cache. If you want to save and reuse the cache files, you can configure the transformation to use a persistent cache. Use this feature when you know the lookup table does not change between session runs. Using a persistent cache can improve performance because the Informatica Server builds the memory cache from the cache files instead of from the database.

Reducing the Number of Cached Rows
Use the Lookup SQL Override option to add a WHERE clause to the default SQL statement. This allows you to reduce the number of rows included in the cache.

Optimizing the Lookup Condition
If you include more than one lookup condition, place the conditions with an equal sign first to optimize lookup performance.

Indexing the Lookup Table
The Informatica Server needs to query, sort, and compare values in the lookup condition columns. The index needs to include every column used in a lookup condition. You can improve performance for both cached and uncached lookups:
Cached lookups. You can improve performance by indexing the columns in the lookup ORDER BY. The session log contains the ORDER BY statement.
Uncached lookups. Because the Informatica Server issues a SELECT statement for each row passing into the Lookup transformation, you can improve performance by indexing the columns in the lookup condition.

Other methods to Improve Performance

Optimizing the Target Database



If your session writes to a flat file target, you can optimize session performance by writing to a flat file target that is local to the Informatica Server.

If your session writes to a relational target, consider performing the following tasks to increase performance:
  • Drop indexes and key constraints.
  • Increase checkpoint intervals.
  • Use bulk loading.
  • Use external loading.
  • Turn off recovery.
  • Increase database network packet size.
  • Optimize Oracle target databases.



Dropping Indexes and Key Constraints


When you define key constraints or indexes in target tables, you slow the loading of data to those tables. To improve performance, drop indexes and key constraints before running your session. You can rebuild those indexes and key constraints after the session completes.

If you decide to drop and rebuild indexes and key constraints on a regular basis, you can create pre- and post-load stored procedures to perform these operations each time you run the session.

Note:

Increasing Checkpoint Intervals
The Informatica Server performance slows each time it waits for the database to perform a checkpoint. To increase performance, consider increasing the database checkpoint interval. When you increase the database checkpoint interval, you increase the likelihood that the database performs checkpoints as necessary, when the size of the database log file reaches its limit.


You can use bulk loading to improve the performance of a session that inserts a large amount of data to a Sybase or Microsoft SQL Server database. Configure bulk loading on the Targets dialog box in the session properties.

If you have indexes or key constraints on your target tables and you want to enable bulk loading, you must drop the indexes and constraints before running the session. After the session completes, you can rebuild them. If you decide to use bulk loading with the session on a regular basis, you can create pre- and post-load stored procedures to drop and rebuild indexes and key constraints. 
For other databases, even if you configure the bulk loading option, Informatica Server ignores the commit interval mentioned and commits as needed.

External Loading on Teradata, Oracle and Sybase IQ

You can use the External Loader session option to integrate external loading with a session.

If you have a Teradata target database, you can use the Teradata external loader utility to bulk load target files.

If your target database runs on Oracle, you can use the Oracle SQL*Loader utility to bulk load target files. When you load data to an Oracle database using a partitioned session, you can increase performance if you create the Oracle target table with the same number of partitions you use for the session.

If your target database runs on Sybase IQ, you can use the Sybase IQ external loader utility to bulk load target files. If your Sybase IQ database is local to the Informatica Server on your UNIX system, you can increase performance by loading data to target tables directly from named pipes. Use pmconfig to enable the SybaseIQLocaltoPMServer option. When you enable this option, the Informatica Server loads data directly from named pipes rather than writing to a flat file for the Sybase IQ external loader.


You can increase the network packet size in the Informatica Server Manager to reduce target bottleneck. For Sybase and Microsoft SQL Server, increase the network packet size to 8K - 16K. For Oracle, increase the network packet size in tnsnames.ora and listener.ora. If you increase the network packet size in the Informatica Server configuration, you also need to configure the database server network memory to accept larger packet sizes.
If your target database is Oracle, you can optimize the target database by checking the storage clause, space allocation, and rollback segments.

When you write to Oracle target databases, the database uses rollback segments during loads. Make sure that the database stores rollback segments in appropriate tablespaces, preferably on different disks. The rollback segments should also have appropriate storage clauses.

You can optimize the Oracle target database by tuning the Oracle redo log. The Oracle database uses the redo log to log loading operations. Make sure that redo log size and buffer size are optimal. You can view redo log properties in the init.ora file.

If your Oracle instance is local to the Informatica Server, you can optimize performance by using IPC protocol to connect to the Oracle database. You can set up Oracle database connection in listener.ora and tnsnames.ora.

When you write to an Oracle database, check the storage clause for database objects. Make sure that tables are using large initial and next values. The database should also store table and index data in separate tablespaces, preferably on different disks.
Optimizing Oracle Target Databases
Increasing Database Network Packet Size
When bulk loading, the Informatica Server bypasses the database log, which speeds performance. Without writing to the database log, however, the target database cannot perform rollback. As a result, the Informatica Server cannot perform recovery of the session. Therefore, you must weigh the importance of improved session performance against the ability to recover an incomplete session.
Bulk Loading on Sybase and Microsoft SQL Server

To optimize performance, use constraint-based loading only if necessary.

PERFORMANCE TIPS IN GENERAL

Most of the gains in performance derive from good database design, thorough query analysis, and appropriate indexing. The largest performance gains can be realized by establishing a good database design.

1. Update Table Statistics in database.
 
SYBASE SYNTAX:

update all statistics table_name

Adaptive Server’s cost-based optimizer uses statistics about the tables, indexes, and columns named in a query to estimate query costs. It chooses the access method that the optimizer determines has the least cost. But this cost estimate cannot be accurate if statistics are not accurate. Some statistics, such as the number of pages or rows in a table, are updated during query processing. Other statistics, such as the histograms on columns, are only updated when you run the update statistics command or when indexes are created.

If you are having problems with a query performing slowly, and seek help from Technical Support or a Sybase news group on the Internet, one of the first questions you are likely be asked is “Did you run update statistics?” You can use the optdiag command (IN SYBASE) to see the time update statistics was last run for each column on which statistics exist:



NOTE:
Running the update statistics commands requires system resources. Like other maintenance tasks, it should be scheduled at times when load on the server is light. In particular, update statistics requires table scans or leaf-level scans of indexes, may increase I/O contention, may use the CPU to perform sorts, and uses the data and procedure caches. Use of these resources can adversely affect queries running on the server if you run update statistics at times when usage is high. In addition, some update statistics commands require shared locks, which can block updates.

• Dropping an index does not drop the statistics for the index, since the optimizer can use column-level statistics to estimate costs, even when no index exists. If you want to remove the statistics after dropping an index, you must explicitly delete them with delete statistics.

• Truncating a table does not delete the column-level statistics in sysstatistics. In many cases, ables are truncated and the same data is reloaded. Since truncate table does not delete the olumn-level statistics, there is no need to run update statistics after the table is reloaded, if the data is the same. If you reload the table with data that has a different distribution of key values, ou need to run update statistics.

• You can drop and re-create indexes without affecting the index statistics, by specifying 0 for the number of steps in the with statistics clause to create index. This create index command does not affect the statistics in sysstatistics

(IN SYBASE):
Create index title_id_ix on titles (title_id) with statistics using 0 values

This allows you to re-create an index without overwriting statistics that have been edited with optdiag.

• If two users attempt to create an index on the same table, with the same columns, at the same time, one of the commands may fail due to an attempt to enter a duplicate key value in ysstatistics.

2. Create Indexes on KEY fields. Keep Index statistics up to date.

NOTE:

If data modification performance is poor, you may have too many indexes. While indexes favor “select operations”, they slow down “data modifications”.
 

ABOUT INDEXES

Indexes are the most important physical design element in improving database performance:

• Indexes help prevent table scans. Instead of reading hundreds of data pages, a few index pages and data pages can satisfy many queries.

• For some queries, data can be retrieved from a nonclustered index without ever accessing the data rows.

• Clustered indexes can randomize data inserts, avoiding insert “hot spots” on the last page of a table.

• Indexes can help avoid sorts, if the index order matches the order of columns in an order by clause.

 In addition to their performance benefits, indexes can enforce the uniqueness of data.

 Indexes are database objects that can be created for a table to speed direct access to specific data rows. Indexes store the values of the key(s) that were named when the index was created, and logical pointers to the data pages or to other index pages.

Adaptive Server (SYBASE) provides two types of indexes:

• Clustered indexes, where the table data is physically stored in the order of the keys on the index:

• For allpages-locked tables, rows are stored in key order on pages, and pages are linked in key order.

• For data-only-locked tables, indexes are used to direct the storage of data on rows and pages, but strict key ordering is not maintained.

• Nonclustered indexes, where the storage order of data in the table is not related to index keys.

You can create only one clustered index on a table because there is only one possible physical ordering of the data rows. You can create up to 249 nonclustered indexes per table. A table that has no clustered index is called a “heap”.

3. Drop and Re-create the Indexes that hurt performance.

Drop Indexes (In Pre-Session) before inserting data AND Re-Create Indexes (In Post-Session) after data is inserted.

NOTE:
With indexes, inserting data is slower.
  
Drop indexes that hurt performance. If an application performs data modifications during the day and generates reports at night, you may want to drop some indexes in the morning and re-create them at night. Drop indexes during periods when frequent updates occur and rebuild them before periods when frequent selects occur.
  

4.  Also you can improve performance by

- Using transaction log thresholds to automate log dumps and “avoid running out of space”.
- Using thresholds for space monitoring in data segments.
- Using partitions to speed loading of data.
 5. To tune the SQL Query

We can use “Parallel Hints” in the SELECT stmt of SQL Query. Also use the table with large no. of rows last when joining. In other sense, use the table with less no. of rows as a MASTER source. Also Queries that contain ORDER BY or GROUP BY clauses may benefit from creating an index on the ORDER BY or GROUP BY columns. Once you optimize the query, use the SQL override option to take full advantage of these modifications.
  

6. Registering Multiple Servers

Also performance can be increased by registering multiple servers which point to same repository.


IMPROVING MAPPING PERFORMANCE - TIPS

1. Aggregator Transformation

You can use the following guidelines to optimize the performance of an Aggregator transformation.

a) Use Sorted

Input to decrease the use of aggregate caches:

The Sorted Input option reduces the amount of data cached during the session and improves session performance. Use this option with the Source Qualifier Number of Sorted Ports option to pass sorted data to the Aggregator transformation.

b) Limit connected input/output or output ports:

Limit the number of connected input/output or output ports to reduce the amount of data the Aggregator transformation stores in the data cache.

c) Filter before aggregating:

If you use a Filter transformation in the mapping, place the transformation before the Aggregator transformation to reduce unnecessary aggregation.

2. Filter Transformation

The following tips can help filter performance:

a) Use the Filter transformation early in the mapping:

To maximize session performance, keep the Filter transformation as close as possible to the sources in the mapping. Rather than passing rows that you plan to discard through the mapping, you can filter out unwanted data early in the flow of data from sources to targets.

b) Use the Source Qualifier to filter:

The Source Qualifier transformation provides an alternate way to filter rows. Rather than filtering rows from within a mapping, the Source Qualifier transformation filters rows when read from a source. The main difference is that the source qualifier limits the row set extracted from a source, while the Filter transformation limits the row set sent to a target. Since a source qualifier reduces the number of rows used throughout the mapping, it provides better performance.

However, the source qualifier only lets you filter rows from relational sources, while the Filter transformation filters rows from any type of source. Also, note that since it runs in the database, you must make sure that the source qualifier filter condition only uses standard SQL. The Filter transformation can define a condition using any statement or transformation function that returns either a TRUE or FALSE value.

  
3. Joiner Transformation

The following tips can help improve session performance:

a) Perform joins in a database:

Performing a join in a database is faster than performing a join in the session. In some cases, this is not possible, such as joining tables from two different databases or flat file systems. If you want to perform a join in a database, you can use the following options:

Create a pre-session stored procedure to join the tables in a database before running the
mapping.
Use the Source Qualifier transformation to perform the join.

b) Designate as the master source the source with the smaller number of records:

For optimal performance and disk storage, designate the master source as the source with the lower number of rows. With a smaller master source, the data cache is smaller, and the search time is shorter. 
  
4. LookUp Transformation
Use the following tips when you configure the Lookup transformation:

a) Add an index to the columns used in a lookup condition:

If you have privileges to modify the database containing a lookup table, you can improve performance for both cached and uncached lookups. This is important for very large lookup tables. Since the Informatica Server needs to query, sort, and compare values in these columns, the index needs to include every column used in a lookup condition.

b) Place conditions with an equality operator (=) first:

If a Lookup transformation specifies several conditions, you can improve lookup performance
by placing all the conditions that use the equality operator first in the list of conditions that appear under the Condition tab. 
 

c) Cache small lookup tables:

Improve session performance by caching small lookup tables. The result of the Lookup query and processing is the same, regardless of whether you cache the lookup table or not.

d) Join tables in the database:

If the lookup table is on the same database as the source table in your mapping and caching is not feasible, join the tables in the source database rather than using a Lookup transformation.

e) Un Select the cache look-up option in Look Up transformation if there is no look up over-ride.
This improves performance of session.