The above command makes Teradata to include additional  info in the output of an explain command.  The output from all subsequent explain commands

• qualify Spool File on join columns in the explain output
• specify hash distribution column names for spool files.

To turn off additional information in the output of the explain command just issue the following command.

DIAGNOSTIC VERBOSEEXPLAIN NOT ON FOR SESSION; 

Below is another command that is quite helpful 

DIAGNOSTIC HELPSTATS ON FOR SESSION;

The above command makes Teradata to include the collect statistics possibilities in the ouput of expplain command

To turn off collect statistics possibilities in the output of the explain command just issue the following command.

DIAGNOSTIC HELPSTATS NOT ON FOR SESSION;

A node contains a large number of hardware and software components as explained below…

System Disks
System disks are contained on the node used for the following:
• Operating system software
• Teradata software
• Application software
• System dump space
Teradata database tables are stored on disk arrays, not on the system disks.

Memory
Vprocs share a free memory pool within a node. A segment of memory is allocated to a vproc for its use, then returned to the memory pool for use by another vproc. The free memory pool is a collection of memory available to the node.

Application
An application is software that accesses the Teradata RDBMS. It can run on various platforms:
• Channel-attached client
• LAN-attached client
• Node

Vproc
A virtual processor or a vproc is a group of one or more software processes running under the operating system's multi-tasking environment:
• On the UNIX operating system, a vproc is a collection of software processes.
• On the Windows operating systems, a vproc is a single software process.

The two types of Teradata vprocs are:
• AMP (Access Module Processor)
• PE (Parsing Engine)

When vprocs communicate, they use BYNET hardware (on MPP systems), BYNET software, and PDE. The BYNET hardware and software carry vproc messages to and from a particular node. Within a node, the BYNET and PDE software deliver messages to and from the participating vprocs.

PE
PEs (Parsing Engines) are vprocs that receive SQL requests from the client and break the requests into steps. The PEs send the steps to the AMPs and subsequently return the answer to the client.

AMP
AMPs (Access Module Processors) are virtual processors (vprocs) that receive steps from PEs (Parsing Engines) and perform database functions to retrieve or update data. Each AMP is associated with one virtual disk (vdisk), where the data is stored. An AMP manages only its own vdisk, not the vdisk of any other AMP.

Vdisk (Virtual Disk)
A vdisk  is the logical disk space that is managed by an AMP. Depending on the configuration, a vdisk may not be contained on the node; however, it is managed by an AMP, which is always a part of the node.
The vdisk is made up of 1 to 64 pdisks (user slices in UNIX or partitions in Windows NT, whose size and configuration vary based on RAID level). The pdisks logically combine to comprise the AMP's vdisk. Although an AMP can manage up to 64 pdisks, it controls only one vdisk. An AMP manages only its own vdisk, not the vdisk of any other AMP.

Channel Driver
Channel driver software is the means of communication between the PEs and applications running on channel-attached (mainframe) clients.

Gateway
The Teradata Gateway software is the means of communication between the PEs and applications running on a LAN-attached clients  or A node in the system

BYNET
The BYNET (banyan network) is a combination of hardware and software that provides high performance networking between the nodes of a Teradata system. A dual-redundant, bi-directional, multi-staged network, the BYNET enables the nodes to communicate in a high speed, loosely-coupled fashion. It is based on banyan topology, a mathematically defined structure that has branches reminiscent of a banyan tree.
The BYNET is a high-speed interconnect (network) that enables multiple nodes in the system to communicate.

The BYNET hardware and software handle the communication between the vprocs.
• Hardware: The nodes of an MPP system are connected with the BYNET hardware, consisting of BYNET boards and cables.
• Software: The BYNET software is installed on every node. This BYNET driver is an interface between the PDE software and the BYNET hardware.

SMP systems do not contain BYNET hardware. The PDE and BYNET software emulates BYNET activity in a single-node environment. The SMP implementation is sometimes called "boardless BYNET."

Teradata uses different types of messages for the highest efficiency:
• Broadcast: BYNET hardware delivers a message to all nodes in an MPP system. This is simulated in SMP systems using BYNET software. A single broadcast message consumes more BYNET bandwidth than a single point-to-point message, but sometimes a broadcast may be the most efficient for processing.
• Point-to-Point: A single message to a single vproc is carried over BYNET hardware to the destination node, and relayed to or from a particular vproc using PDE software. Point-to-point messages enable the BYNET to be scalable because as the system grows, the capacity to handle point-to-point messages also grows.
• Multicast: A message can be sent to several vprocs (multicast). The BYNET hardware first sends a broadcast message to all nodes. The PDE software on the receiving nodes determines which vprocs should receive the message and delivers it to those vprocs. Multicast messages give the system yet another method to deliver messages for the best performance
 
A Teradata node requires three distinct pieces of software.

Operating System : 
UNIX or Windows. Or Linux

The Parallel Database Extensions (PDE) :
The Parallel Database Extensions (PDE) software layer was added to the operating system by NCR to support the parallel software environment.

Teradata RDBMS aka Trusted Parallel Application (TPA)
A Trusted Parallel Application (TPA) uses PDE to implement virtual processors (vprocs). The Teradata RDBMS is classified as a TPA. The four components of the Teradata TPA are:
• AMP
• PE
• Channel Driver
• Teradata Gateway
 
Access Module Processor (AMP)
AMPs (Access Module Processors) are virtual processors (vprocs) that receive steps from PEs (Parsing Engines) and perform database functions to retrieve or update data
AMP Worker Task Functions
The AWT functions in the AMP perform a number of operations, including:
• Locking tables to ensure data consistency.
• Executing AMP step operations such as select, insert, update, delete and sort.
• Joining tables as required.
• Executing end transaction steps as required to support multi-AMP operations.

AMP File System
The file system software accesses the data on the virtual disks. Each AMP uses the file system software to read from and write to the virtual disks.

AMP Console Utilities
The AMP software includes utilities to perform systems management functions such as:
• Configure and reconfigure the system
• Rebuild tables
• Reveal details about locks and space status
 
Parsing Engine (PE)
PEs (Parsing Engines) are vprocs that receive SQL requests from the client and break the requests into steps. The PEs send the steps to the AMPs and subsequently return the answer to the client.

PE Session Control 
When you log on to the Teradata RDBMS through your application, the session control software on the PE establishes that session. Session control also manages and terminates sessions on that PE.

PE Parser/Optimizer
The Parser interprets an incoming Teradata SQL request and checks the syntax. The Parser decomposes the request into AMP steps, using the Optimizer to determine the most efficient way to access the data on the virtual disks (vdisks).
The Optimizer develops the "least expensive" plan (in terms of time and system resources) to return the requested response set. Processing alternatives are evaluated, and the fastest alternative is chosen. The selected alternative is converted to executable steps that will performed by the AMPs. Then, the Parser sends the steps to the dispatcher.

PE Dispatcher
The dispatcher is responsible for a number of tasks, depending on the operation it is performing:
• Processing Requests: Controls the sequence in which the steps are executed and passes the steps to the AMPs through the BYNET.
• Processing Responses: After the AMPs process the steps, the dispatcher builds a response message and sends the response back to the user. 

Channel Driver
Channel Driver software is the means of communication between an application and the PEs assigned to channel-attached clients. There is one Channel Driver per node.

Teradata Gateway
Teradata Gateway software is the means of communication between an application and the PEs assigned to network-attached clients. There is one Teradata Gateway per node.
 
User Access
Clients communicate with the Teradata RDBMS in three ways:
• Channel
• Network
• Node

Channel
Communication from applications on the mainframe goes through the Channel Driver.
Traffic and communications between client applications on the mainframe and the Teradata database are managed by the Teradata Director Program (TDP) software.

Network
Communication from applications on a network-attached client goes through the Teradata Gateway.
Traffic and communication between network-attached client applications and the Teradata database are managed by either: ODBC or CLIv2.

Node
If you install application software on a node, it will be treated like an application on a network-attached client. In other words, communications from applications on the node go through the Teradata Gateway.

To summarize, users can access data in the Teradata RDBMS through an application on both channel-attached and network-attached clients. Additionally, the node itself can act as a client. Teradata client software is installed on each client (channel-attached, network-attached, or node) and communicates with RDBMS software on the node.
 
Primary Indexes
The Teradata Database distributes the data in each table  across all AMPs on a system.  Since each amp is responsible for managing it’s own vdisk, it effectively means distributing data across disks automatically. This is one of the major differences between Teradata and other RDBMSs

Teradata assigns rows to AMPs based on the value of their primary index. The determination of which hash bucket, and hence which AMP the row is to be stored on, is made solely on the row hash value of its primary index. Each Teradata Database table must have a primary index.

• Brief history and overview of SQL
• Set-up environment variables to invoke sqlplus and log on to a Oracle database from a reflection session
• Execute basic SQL queries
• Get all the data in a table
• Get named columns from a table
• Get named columns from a table and order them
• Perform arithmetic on column values
• Prevent duplicate row selection
• Selecting only data that meets a criteria with the WHERE clause
• SQL operators in WHERE clauses
• Multiple condition WHERE clauses
• Group Functions
• Sub-queries
• Select data from multiple tables
• Execute basic SQL DML commands
• Insert
• Update
• Delete
• Use basic DDL commands
• create table
• alter table
• create index
• truncate table
• drop table
• Use basic access control commands
• grant
• revoke
• Modify sqlplus environment
• Write and execute small SQL scripts
 
Structured Query Language

Structured Query Language or SQL was originally developed by IBM for its System/R Project. IBM later used it in its products SQL/Data System (SQL/DS) and Database 2 (DB2). SQL today became a language of choice to access the data and structures within a relational database and is implemented and supported by most commercially available Relational Database Management System vendors.

SQL was designed to be an English like language using English phrases to manipulate the database. It is non procedural. You specify what you want done but not how to do it. The how part is taken care of by the RDBMS. Each RDBMS has an in-built query optimizer which parses your SQL statements and works out the best path to retrieve/manipulate the data.

SQL is designed to be used by a wide spectrum of users- DBA's, application developers, operators  and end users.

It provides commands to do the following …

• finding (querying) data
• inserting, updating and deleting data
• creating, modifying and deleting database objects
• controlling access to the database and database objects
• Transaction Control

The different SQL statements could be  grouped into 4 different categories as explained below…

DATA QUERY LANGUAGE (DQL) -  SQL statements  to retrieve data from the database and transform it.   SELECT statement falls into this category.

DATA MANIPULATION LANGUAGE (DML) - SQL statements to insert, change or delete data from the database. DELETE, INSERT, UPDATE   statements  fall into this category.

DATA DEFINITION LANGUAGE  (DDL) - SQL statements to define or alter database structures. DROP, TRUNCATE, CREATE, ALTER  statements  fall into this category.

Access Control - SQL statements to either grant or revoke privileges to access and manipulate  database objects. 

REVOKE, GRANT statements  fall into this category.

Transaction Control - SQL statements that commit or rollback changes to data.

COMMIT, ROLLBACK statements fall into this category.

To connect to an oracle database and execute sql commands you can use an oracle supplied tool called sqlplus.
 
Environment set-up for invoking sqlplus and connect to an Oracle  database

Add the following lines to your .profile after you login to your unix environment. After adding the following lines log out and log back in to make sure that the .profile is run.

# Replace with the path at your installation
export ORACLE_HOME=/oracle/product/8.1.7
# Replace with sid name at your installation
ORACLE_SID=xyz
export ORACLE_SID
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
export LD_LIBRARY_PATH
LIBPATH=$LIBPATH:/usr/lib:$ORACLE_HOME/lib
export LIBPATH
PATH=$PATH:$ORACLE_HOME/bin
export PATH

The above lines set-up paths for oracle binaries and libraries.

The ORACLE_SID environment variable identifies the database that you would connect to when you invoke sqlplus. In our case we would be connecting to a database named stgt.

After setting up the environment you can invoke sqlplus by entering sqlplus at your unix prompt as shown below. Enter your username and password when prompted. You will get SQL prompt if you login successfully. You can then enter and execute all your sql commands at the SQL prompt.

$ sqlplus

SQL*Plus: Release 8.1.7.0.0 - Production on Mon Aug 26 11:33:41 2002

(c) Copyright 2000 Oracle Corporation.  All rights reserved.

Enter user-name: rnemani
Enter password:

Connected to:
Oracle8i Enterprise Edition Release 8.1.7.4.0 - Production
With the Partitioning option
JServer Release 8.1.7.4.0 - Production

SQL>

If you want to terminate your sqlplus session you simply enter exit at your SQL prompt as shown below…

SQL> exit
Disconnected from Oracle8i Enterprise Edition Release 8.1.7.4.0 - Production
With the Partitioning option
JServer Release 8.1.7.4.0 – Production

$

DATA QUERY LANGUAGE (DQL)

The SELECT statement is the most widely used command of SQL. It allows you to retrieve the data from the database. Most SQL statements will need a semi-colon ";" to indicate the end of the statement.

Get all the data  in a table

Select * from price_list ;

Gets all the columns and all the rows in the table.

Get named columns from a table

Select  division_no, article_nbr, price_list_price, price_list_price
From price_list ;

Get named columns from a table and order them

Select  division_no, article_nbr, price_list_price
From price_list
Order by division_no, article_nbr ;

Perform arithmetic on column values

Select  division_no, article_nbr, price_list_price, price_list_price*0.05
From price_list ;

Preventing duplicate row selection

Select  distinct division_no From price_list ;

Selecting only data that meets a criteria  with the WHERE clause

Select  division_no, article_nbr, price_list_price
From price_list
Where division_no = 72 ;

SQL operators in WHERE clauses

Select  division_no, article_nbr, price_list_price
From price_list
Where division_no in  (71, 72)  ;

Select  division_no, article_nbr, price_list_price
From price_list
Where price_list_price > 100 ;

Select  division_no, article_nbr, price_list_price
From price_list
Where division_no  not in  (71, 72)  ;

select * from division
where division_desc like 'T%' ;

Multiple condition WHERE clauses

Select * from price_list
Where division_no = 93
And basis_company_id = 66 ;

Select * from price_list
Where division_no = 93
or price_list_price > 100 ;

Group Functions

Select count(*) from price_list ;

Select max(price_list_price)
From price_list ;

Select avg(price_list_price)
From price_list ;

Select division_no, count(*)
from price_list
group by division_no ;

Select division_no, max(price_list_price), min(price_list_price)
from price_list
group by division_no ;

Select division_no, max(price_list_price), min(price_list_price)
from price_list
group by division_no
having min(price_list_price) > 10 ;

Select division_no, basis_company_id, count(*)
from price_list
group by division_no, basis_company_id ;

Selecting data from multiple tables (Join)

Select division_desc, outlet_name
From division a , outlet b
Where a.division_no = b.division_no ;

Select division_desc, outlet_name
From division a, outlet b
Where a.division_no = b.division_no
And a.division_no = 51 ;

Sub-queries

Select count(*) from price_list
Where division_no in (
select division_no from division where price_group_user = 'Y'
) ;

Select count(*) from price_list
Where division_no not in (
select division_no from division where price_group_user = 'Y'
)
;

Select count(*) from price_list  a
Where exists in (
select division_no
from division b
where b.price_group_user = 'Y'
and b.division_no = a.division_no
)
;

Select count(*) from price_list  a
Where not exists in (
select division_no
from division b
where b.price_group_user = 'Y'
and b.division_no = a.division_no
)
;
DATA MANIPULATION LANGUAGE (DML)

In order for us to be able to retrieve data from tables we need to first have them populated and properly maintained so that they contain the relevant data. This is exactly what the DML commands are used for.

INSERT

Insert command is used for inserting new data into a table.

insert into division
values (98, 'This is a New Dvsn','Y','Y','N')
;

The above statement will create a new row .

insert into division
(division_no,division_desc)
values (98, 'This is a New Dvsn') ;

The above statement will create a new row but with only division_no and division_desc columns populated with values and other columns set to NULL.

Insert into new_division_table
Select * from division ;

The above statement Inserts rows from division table into new_division_table table (provided the structure of new_division_table table is identical to the structure of division table).

You have to issue a commit statement if you want the data to be saved in the table permanently. Alternatively you can issue a rollback, if you do not want your changes to be saved to the database.

A commit or a rollback command should always be issued to commit or rollback a transaction. A transaction could be made up of one or more DML commands.

A commit statement looks as follows …

Commit ;

A rollback statement looks as follows …

Rollback ;

Make sure that you issue a commit or rollback before you exit your sqlplus session. Sqlplus will issue a commit by default when you exit.

UPDATE

The UPDATE command is used for updating the column values of an existing row in a table.

Following are some examples …

update division
set division_desc = 'My new division'
where division_no = 98 ;

update price_list
set price_list_price = price_list_price * 0.05
where price_list_price < 1 ;

You can use a sub-query to update a column as shown below …

update new_division_table a
set division_desc = (select division_desc from division b
                     where b.division = a.division
                     )
;

DELETE

Delete command is used for removing data (rows) from a table.

delete from  new_division_table ;

The above statement will delete all rows from new_division_table table.

delete from  new_division_table
where division_no = 98 ;

The above statement will delete the row corresponding to the division_no 98 from the new_division_table table.

Some more examples …

delete from  new_division_table
where division_no >= 98 ;

delete from  new_division_table
where division_no in (98, 99) ;

delete from  new_division_table
where division_no != 98 ;

delete from  new_division_table
where division_no not in (98, 99) ;

DATA DEFINITION LANGUAGE (DDL)

In order for us to be able to store and retrieve data from tables we need to first have them created. This is exactly what the DDL commands are used for.

Following is an example of "create table " command in its simplest form…

create table new_division_table
( DIVISION_NO  NUMBER(2) NOT NULL,
DIVISION_DESC  VARCHAR2(30),
PRICE_GROUP_USER  VARCHAR2(1),
TRADE_GROUP_USER  VARCHAR2(1),
SALES_TYPE_USER  VARCHAR2(1)
) ;

The above command creates a table called new_division_table. It will be empty. The NOT NULL clause  for division_no column is to make sure that division_no column must always have a value whenever data is inserted into this table and it can not be updated to a NULL value.

You can also create a table from an existing table as shown below…

create table new_division_table as
select * from division ;

The above command creates a table called new_division_table and populates it with data from division table.

create index x1_new_division_table
on new_division_table (division_no) ;

The above command creates an index called x1_new_division_table.

Alter table  new_division_table
modify               DIVISION_DESC  VARCHAR2(40) ;

The above command alters the width of division_desc column to 40 from 30.

truncate table new_division_table ;

The above command deletes all data in the table

drop table new_division_table ;

The above command drops the table from the database. All the data within the table is permanently lost. So do exercise caution when using this command. I have one suggestion to avoid mistakes with DROP command – DO NOT USE IT in the first place.

Note : DDL commands do not need a commit because they issue an implicit commit. This means that we can not rollback any DDL once it is executed.

There are a number of other DDL commands which I think are the beyond the scope of this brief introduction to SQL.
 
Access Control Commands

When a user creates a database object such as a table, it gets created under the ownership (schema) of the user creating it. The user who created the table owns the table.  In order for others to be able to access or store data in this table the owner has to grant appropriate privileges to other users.

Following are some examples but first the disclaimer…

"Following examples are based on fictitious situations. Any resemblance to any names of any persons or projects or places is purely coincidental. " 

Example 1.

Tom and Harry work in the IT department for a major bottling company and are working on a new project (actually cleaning up an old project) which requires them to work on an Oracle database and deal with tables etc...

During the course working on this project, Tom creates a table called new_division_table and he wants Harry to be able to select data from this table. So he executes the following command…

grant select on new_division_table to Harry ;

Now Harry can access data from this table as follows..

select *  from Tom.new_division_table ;

Note that the username of the owner of the table has to be prepended to the table name, if the table is not owned by the user trying to access the table.

Example 2.

Harry tells Tom that he wants to be able to do more than mere selecting of data from the table.

Tom then executes the following GRANT command to give Harry more privileges…

grant insert, update, delete on new_division_table  to Harry ;

Harry then tests his newly granted privileges on the table as follows…

delete from  Tom.new_division_table ;
commit ;

Example 3.

Now Tom is mad and wants to take away all but SELECT privilege from Harry on the new_division_table. He executes the following command …

revoke   insert,update, delete on new_division_table from Harry ;

I Hope the above examples clarify the use of Access control commands.

Changing sqlplus environment

You can change the default behaviour of how the results are displayed within sqlplus. You can do this by using the set command of sqlplus.

set pagesize 80

The above command makes sure columns headings are repeated  after every 80 lines of the results.

set pagesize 0

The above command supresses printing of headings.

set linesize  70

Above command sets the line size to 70.

spool /home/rnemani/division.list
select * from division ;
spool off

The above commands create a text file called division.lst in /home/patuser directory. This file would contain the output of division table.

If you want see your current sqlplus environment settings give the following command

show all

There are a number of useful set commands that you could use and would use as you start using sqlplus regularly.

Using SQL scripts

You can save regularly used sql commands in a sql script using a text editor such as vi.

For example you can save the following commands in a  script file called get_div.sql  using vi.

spool /home/rnemani/division.list
select * from division ;
spool off

You can then start up a sql session and execute the script at the sql prompt as shown below.

SQL> @get_div.sql

The above command would execute all the commands in the script as if they were entered at SQL prompt.

You can know the structure (column names, data types etc…) of any table, by using the following command

DESC table_name

for example…

SQL> desc division
 Name                                Null?    Type
 ----------------------------------- -------- --------------------
 DIVISION_NO                         NOT NULL NUMBER(2)
 DIVISION_DESC                                VARCHAR2(30)
 PRICE_GROUP_USER                             VARCHAR2(1)
 TRADE_GROUP_USER                             VARCHAR2(1)
 SALES_TYPE_USER                              VARCHAR2(1)

SQL>

Oracle returns errors when you execute your sql statements, if there is a syntax error(s) or if it is unable to find the objects that are referred to in the SQL statement. Given below is an example…

SQL> select count(*) from rkn_branch_extension ;
select count(*) from rkn_branch_extension
                     *
ERROR at line 1:
ORA-00942: table or view does not exist

If you want know more about this error then you can issue the following command at your UNIX prompt.

$ oerr ORA 942

Conclusion

As you can see SQL is very easy to use and intuitive. One other good thing about SQL is that it is portable for the most part across databases that support it. I hope this tutorial gave you enough information to get started with SQL.

In part 2 of this article I cover writing Efficient SQL Code for the OLTP environment.

This article aims to provide you with information and tips that will help you write efficient SQL code for Oracle databases. Part 1 covers batch environment.

Why is writing efficient SQL code important?
If you would like your application to scale well, then you would need to have SQL code that executes efficiently.  

How do you define efficient SQL code?
Any SQL code that does what it does, using minimum resources and in minimum possible time.

Huge Inserts
Huge Updates
Huge Deletes
Efficient elimination of duplicates
Efficient verification of  foreign keys

Oracle does the following when you issue a DML
After the normal syntax, semantics and security checking etc...
it would...
     1. Insert/modify the data into database buffers
     - This is the actual data that would subsequently be written to data  files on the disks
     2. Create redo log records
     - These records are written to redo log files
     - These are used to recover from an instance failure or crash
     3. Create undo records (Rollback info)
     - These records are written to rollback segments
     - These records are used in case you decide to rollback

So, as you can see each DML, results in a flurry of activity which means batch jobs that deal with huge amounts of data would have unacceptable performance unless special measures are taken as described next.

Handling huge inserts
Normal insert
SQL> insert
2  into abc
3  select * from xyz  ;

 978816 rows created.

Elapsed: 00:08:18.08

Statistics
-------------------------------------------------
  106586580  redo size

 Obviously the above insert is not very efficient. It took about 8+ minutes to insert just under a million rows.

Now let us look at a turbo charged version of insert command called  Direct Load Insert

SQL> insert /*+ append */
2  into abc
3  select * from xyz
4  nologging  ;

978816 rows created.

Elapsed: 00:00:12.30

Statistics
----------------------------------------------------------
     241716  redo size

“Direct load insert” works differently than a Normal insert. It completely by passes the 3 steps (mentioned previously) that occur for a normal DML and directly writes to the data files of the oracle database.

You should use “Direct load insert” whenever you do huge inserts.

Handling large Updates

Say you have a 30 million row table and your update affects 5 Millions rows - How would you do it ?
The normal approach  would be to execute an update statement similar to the one shown in the example below

update  aBigTable
set column1 = column1 + column1 * 0.10
where column3 = 'Y'
;

What do you think will happen if you executed the previous update?

Best case scenario
1. Your update finishes after running zillion hours without actually running out of rollback segment space
2. Because your update brought everything else to a grinding halt  - Help desk gets inundated by calls from other users of the database wanting to know if the database has left the building!!!.

Worst case scenario
1. Your update takes zillion hours and then fails after running out of rollback segment space
2. Because your update brought everything else to a grinding halt  - Help desk gets inundated by calls from other users of the database .
3. Your access privileges to the database may get revoked
As you can see even the best case scenario is not particularly good…
let us look at a better way of doing large updates
--
-- Create an empty table
--
create table aBigTable_temp
nologging
as select * from aBigTable where rownum < 1 ;
insert /*+ append */
into aBigTable_temp (column1,column2,column3)
select decode (column3, 'Y', (column1 + column1 * 0.10), column1)
, column2
, column3
from aBigTable ;
commit ;
rename  aBigTable to aBigTable_old ;
rename  aBigTable_temp  to aBigTable ;
--
-- Notice how column1 is handled in the select clause of the insert ..select .. statement
--
--
-- create any needed indices on aBigTable
--
Your index creation statements go here.
--
-- Check if data in table aBigTable is OK and drop aBigTable_old
--
drop table aBigTable_old ;

The above steps would complete in far less time than it takes the normal update

Handling large Deletes
Say you have a 30 million row table and you need to delete  5 Millions rows - How would you do it
The normal approach  would be to execute the following sql statement
 
Delete  from xyz
where column3 = 'Y‘ ;

What do you think will happen ?
The best and worst case scenarios would be more or less the same as that for an update described previously.

Recommended approach
--
-- Create an empty table
--
create table temp_table
nologging
as select * from xyz where rownum < 1 ;
--
-- Load the rows that you want to keep in the temp table
--
insert /*+ append */
into temp_table (column1,column2,column3)
select column1 , column2 , column3
from xyz
where column3 != 'Y‘  ;
commit ;

rename xyz  xyz_old  ;
rename xyz_temp  xyz  ;
--
-- create any needed indices on xyz
--
Your index creation code goes here.
--
-- Check if data in table xyz is OK and drop xyz_old
--
drop table xyz_old ;

Efficient Duplicates elimination

Many times we are faced with situations that require us to design a load script that loads data with duplicates.

Normal method of eliminating duplicates
1.Define a unique index on the table  and load data.
2.Load the data using sqlldr conventional path (direct path load can not  be used because of the index)
sqlldr rejects Duplicate rows and writes them to badfile. But this unique key checking incurs a very big performance penalty. For example in my testing sqlldr took 28 minutes to load  little over a million rows  with one duplicate row. Using the normal method, the time it takes to eliminate duplicates grows exponentially with the size of data. So this method is not practical for bulk data loading.

Method to eliminate duplicates efficiently
Oracle assigns a unique rowid to each row in a table.   So each row in a table has unique rowid even if the data within the row is duplicated.
In oracle the fastest path to data is the path that uses rowids. So if we can identify the rowids of duplicates then we can use those rowids to delete them from the table very fast.
The sample delete statement  given  next , identifies the duplicates and deletes them.  

This sample delete statement is explained using an  example in subsequent slides

delete  from xyz
where     rowid in
(
--
-- This  select gets all the rowids
--
     select    a.rowid
     from      xyz
     minus
--
-- This select gets  rowids of the first row for each duplicated value for the
-- column in the group by clause
--
     select    min(rowid)
     from      xyz
     group by unique_key_col
) ;

The minus operation gives us the  row ids of all the duplicates that can be  deleted from the table. Thus, we  will have only one row for each set of duplicates in the table after the deletion thus ridding the table of duplicates.

I use a table called emp_sal to explain the above delete statement whose contents are shown below
    EMPID DEPTID          SAL
 ---------- ----------     ----------
         1 SALES           36000
         2 SALES           38000
         3 SALES           42000
         4 ACCTS           36000
         5 ACCTS           38000
         5 ACCTS           42000
         5 ACCTS           42000
         5 ACCTS           38000
         7 SALES           42000
         7 SALES           35000
         9 IT              42000
        10 IT              45000
        11 IT              52000
        12 IT              55000
        13 SALES           35000
15 rows selected.

This gives all rows including duplicates

SQL> select a.rowid , a.empid from emp_sal a
  2  order by empid   ;
ROWID                         EMPID
------------------               ----------
AAAYPOAAJAABio4AAA          1
AAAYPOAAJAABio4AAB          2
AAAYPOAAJAABio4AAC          3
AAAYPOAAJAABio4AAD          4
AAAYPOAAJAABio4AAE          5
AAAYPOAAJAABio4AAF          5
AAAYPOAAJAABio4AAO          5
AAAYPOAAJAABio4AAN          5
AAAYPOAAJAABio4AAG          7
AAAYPOAAJAABio4AAH          7
AAAYPOAAJAABio4AAI          9
AAAYPOAAJAABio4AAJ         10
AAAYPOAAJAABio4AAK         11
AAAYPOAAJAABio4AAL         12
AAAYPOAAJAABio4AAM         13
15 rows selected.

The following select statement  gives all rows but with only one row per duplicated data

SQL> select min(a.rowid) min_rowid, a.empid
  2  from emp_sal a
  3  group by empid    ;

MIN_ROWID                EMPID
------------------             ----------
AAAYPOAAJAABio4AAA          1
AAAYPOAAJAABio4AAB          2
AAAYPOAAJAABio4AAC          3
AAAYPOAAJAABio4AAD          4
AAAYPOAAJAABio4AAE          5
AAAYPOAAJAABio4AAG          7
AAAYPOAAJAABio4AAI          9
AAAYPOAAJAABio4AAJ         10
AAAYPOAAJAABio4AAK         11
AAAYPOAAJAABio4AAL         12
AAAYPOAAJAABio4AAM         13
11 rows selected.

If  we do a diff (minus operation) on the above two sets of data  then we get the rowids of the duplicates which we use for deleting.

SQL> select a.rowid , a.empid from emp_sal a
  2  minus
  3  select min(a.rowid) , a.empid
  4  from emp_sal a
  5  group by empid
  6  order by empid   ;

ROWID                        EMPID
------------------              ----------
AAAYPOAAJAABio4AAF          5
AAAYPOAAJAABio4AAN          5
AAAYPOAAJAABio4AAO          5
AAAYPOAAJAABio4AAH          7
4 rows selected.

Here is the delete (With all components combined) …

SQL> delete from emp_sal
  2  where rowid in
  3  (
  4  select a.rowid
  5  from emp_sal a
  6  minus
  7  select min(a.rowid)
  8  from emp_sal a
  9  group by empid
 10  )   ;
4 rows deleted.

So in this method you

1. Load the table using sqlldr direct path
2. Use a delete statement similar to the one shown on the previous slide  to eliminate duplicates

Using this method I repeated the test of loading  a little over million rows  with one duplicate row

1.sqlldr took 58 seconds to load 1065393 rows
2.delete statement took 46 seconds.

So it took less than 2 minutes to  load the table and eliminate the duplicates

Efficient Verification of Foreign Key

This is another area that causes performance problems in batch environment.
If you have a  foreign key constraint defined for a column in a table, then all the inserts and updates cause Oracle to do a foreign Key checking to make sure that referential integrity is maintained.  This means for each insert, Oracle would query the referenced table (parent table) to make sure FK constraint is satisfied. This does not pose a problem if you are dealing with low volumes of data. But this becomes a big issue if you do huge data loads.

So how do you  load millions of rows into a table that has foreign keys defined ?

Following two methods are the most widely used but inefficient methods.
Do the load with foreign keys enabled and only rows that satisfy  fk constraints get loaded
or
Do the load  with foreign keys disabled.

Then try enabling the fk with a statement similar to the one below

ALTER TABLE emp
   ENABLE VALIDATE CONSTRAINT fk_deptno
   EXCEPTIONS INTO exceptions_table

Both of the above methods are very inefficient and would require  big batch windows.

The following method is very efficient for loading tables with FK constraints. It involves 3 steps…
1. Load the table with FK disabled (This will be quick since no FK checking is done)
2. Identify the rows that do not satisfy the FK constraint and delete them
3. Enable foreign key without validating (This will be quick since no FK checking is done)
Following example illustrates the use of  the above method.

Desc emp
 Name            Type
 --------------  ------------
 EMPNO           NUMBER(6)
 ENAME           VARCHAR2(10)
 JOB             VARCHAR2(9)
 MGR             NUMBER(6)
 HIREDATE        DATE
 SAL             NUMBER(7,2)
 COMM            NUMBER(7,2)
 DEPTNO          NUMBER(2)

Desc dept
 
Name            Type
 --------------  -------------
 DEPTNO          NUMBER(2)
 DNAME           VARCHAR2(14)
 LOC             VARCHAR2(13)

Deptno in EMP table references Deptno in DEPT table.
So any new rows inserted into EMP table should have a DEPTNO value that exists in DEPT table.
Now to do a huge data load into EMP table
First Load the data into EMP table with FK disabled.
Then Delete from EMP table deptno values that do not exist  in DEPT table    as shown next
delete from emp
where rowid in
(-- This gets us the rowids that need to be deleted
     select rowid
     from emp
     where dept_no in 
     ( -- This identifes deptno values that do not exist in DEPT table
          select distinct deptno
          from emp
          minus
          select deptno
          from dept
     )
) ;

After the above step, You can  re-enable the foreign key constraint on EMP table    without validating. This is because we know that EMP table will have only    valid deptno values after the above delete

ALTER TABLE emp
   ENABLE NOVALIDATE CONSTRAINT fk_deptno ;

This concludes part 1. I have used all the techniques that were described here to make Oracle batch environment perform optimally. I hope you find these techniques useful.