Professional Documents
Culture Documents
The last couple articles I have written focused on meta-data or DDL extraction for Oracle. The search for a part III to those
articles lead me to Oracle's Data Pump utility. Not necessarily for the data movement piece but because it has an API for
meta-data. Well even though I have been using 10g for quite some time, I have yet to use Data Pump. I thought this would be
a great way to introduce myself, and possibly you the reader, to this new utility. This article will serve as a basic introduction
to Data Pump and then in subsequent articles we will walk through the new command line options for Data Pump's export and
import (expdp & impdp), and look at the PL/SQL packages DBMS_DATAPUMP and DBMS_METADATA.
These to command line utilities are very close to the old standby export & import (exp & imp) utilities. They are not stand-
alone utilities in the sense that they use the DBMS_DATAPUMP PL/SQL package to execute the export and import functions.
They accept a variety of command line options that, like exp & imp, allow you to pick and choose the objects to be exported
and imported.
DBMS_DATAPUMP
The Data Pump API and can be used independently of expdp & impdp. Is the package accessed to move data and / or
metadata between databases.
DBMS_METADATA
The meta-data API in Oracle and can also be used independently of expdp & impdp. If you remember this is the package we
were using in the last two articles for extracting meta-data. I am very interested in how it interfaces with Data Pump.
- DBA_DATAPUMP shows all the active Data Pump jobs and details on the state of the job.
- DBA_DATAPUMP_SESSIONS shows the user sessions that are attached to a Data Pump job.
- V$SESSION_LONGOPS shows the progress of a Data Pump job in regards to the amount of work it needs to do and
how much it has done.
o Through the use of internal tables, and what Oracle calls master & worker processes, there exists an
intercommunication between the processes that are performing work on behalf of Data Pump and internal logging
information that allows the user, and Oracle, to understand where in the process Data Pump is.
My security concerns
o Having EXP_FULL_DATABASE & IMP_FULL_DATABASE privileges opens up users being able to see too much and affect
too many objects across the database where you may want a higher granularity of security.
o Operations across the network. This has implications that allow for unsecured network nodes to be infiltrated where
your production servers could then be accessed from and then compromised.
o By definition, Oracle gives permission to the objects in a DIRECTORY that a user would not normally have access to.
Also be aware that there is a default server-based directory object, DATA_PUMP_DIR that Data Pump will look for if a
DIRECTORY is not specified. While I have not seen this directory created by default in an Oracle installation and the
manuals say it needs to be created by a DBA, it is still something to be on the look out for in subsequent releases as
it could cause a security hole.
o Couple the possibility of network holes with the ability to spawn multiple client (expdp & impdp) services, a server
could easily be overcome with processes that would either become a bottleneck or bring your system to a halt.
o It is highly recommended that new monitoring and auditing of these new server processes should be undertaken or at
least validating that they are shut down or restricted to limited use.
Data Pump seems to have a lot of nice features to facilitate the movement of data and meta-data between databases. The
monitoring and tuning of the Data Pump jobs seems to be the greatest benefit to moving to this new form of export and
import. My only concern is the opening up of the database to see the O/S through database links and directory objects.
Hopefully this article served as a nice introduction to Data Pump, I know I learned a lot just by writing this high level
overview. Come back for subsequent parts to this introduction and we will explore how Data Pump really works and hopefully
put to bed any security concerns of mine.
Since we are all familiar with Oracle’s original export (exp) utility, and in my opinion Data Pump will be replacing exp soon, I
thought it would be good to start off getting familiar with this utility by some relatively simple Data Pump exports (expdp)
that are similar to the way we have used exp in the past. In particular the FULL export.
As a word of caution, the Data Pump exports (expdp) are not compatible with exp. So as you go forth and play with this
utility please at least name the exports something that will signify that the dump file was created by expdp so that you won’t
get confused. Also since this utility is not backward compatible, if you have any databases prior to 10g and you are using exp,
you may want to hold off on implementing the new expdp utility as you will not be able to import into any pre-10g databases.
Oracle creates dump and log files through DIRECTORY objects. So before you can use Data Pump you must create a DIRECTORY
object. There are a few different “default” mechanisms for Oracle to determine an appropriate DIRECTORY to use. Mostly
through environment variables and a default directory name that Oracle will look for. But as we all should know, we should
not leave this to chance and instead explicitly create and use a directory object name of our choice. As soon as you create an
object, a DIRECTORY here, that is a default you open yourself up to security breaches and thus this practice should be
avoided. So for here I have logged into my S1 database and will create a DIRECTORY named datapump.
Then, as you use Data Pump you can reference this DIRECTORY as a parameter for export where you would like the dump or
log files to end up. It is good to note here that as dump and log files are created, log files that are written to will overwrite
existing log files of the same name but dump files that have the same name will only create an error condition and error out
the Data Pump job. This was not the case with Oracle’s original export utility (exp). Subsequent exports would overwrite all
files. With Data Pump this is a nice safeguard but can also create problems for those of us who did nightly exports to the
same location and file names. Now we have to think about cleanup routines. A small price to pay for additional security that
could save your life one day when the scraping utility fails.
Just like the original exp utility Data Pump requires some authorization to allow users to export. Here I am granting
EXP_FULL_DATABASE to a user JKOOP on database S1 that will allow the user to perform a full database export. If not given
the JKOOP user could only export their own schema. Also I need to grant READ and WRITE privileges on the recently created
DIRECTORY. Also on database S1.
We have also used the exp utility to connect through a TNS entry to perform an export on a remote database. Data Pump can
also easily do this by adding a connection identifier to the user/password parameter. The exact same way done in exp.
Now for a few export trickeries. These next two examples assume an additional database named S2. They allow for a
connection to the target database that we want to export through a database link. So the first thing to do is create a
database link.
SQL-S2> CREATE DATABASE LINK S1 CONNECT TO JKOOP IDENTIFIED BY PWD USING 'S1';
The key item to remember with Data Pump and where files will end up is the fact that wherever you Data Pump runs it
requires a DIRECTORY to place dump and log files in. So since we will be connecting to the S2 database there will be required
a DIRECTORY for placing these files in. Here I create a new DIRECTORY named mydump on database S2.
Now for the command line options. Here we are running on the server where database S2 resides and will be producing a full
dump of database S1 through the NETWORK_LINK. But placing the dump and log files on the server where database S1
resides. This was great news for me as when I first read the documentation I thought all dumps would have to reside on the
server the database resided on. Now I can almost produce an environment where a single database is a ‘gateway’ for my
database exports if needed.
Ok, suppose we do produce that gateway for exports. Do we need to execute all commands from that server? No! With Data
Pump we need only connect to the S2 database through a TNS entry and then supply the appropriate NETWORK_LINK to the
database we want to export.
DBA_DATAPUMP_JOBS
This view will show the active Data Pump jobs, their state, degree of parallelism, and the number of sessions attached.
DBA_DATAPUMP_SESSIONS
This view give gives the SADDR that assist in determining why a Data Pump session may be having problems. Join to the
V$SESSION view for further information.
V$SESSION_LONGOPS
This view helps determine how well a Data Pump export is doing. Basically gives you a progress indicator through the
MESSAGE column.
The original export utility (exp) may or may not be going away soon. The documentation clearly states that Data Pump will
handle data types that exp will not and we should begin our migration to this new utility. Except for those instances where
you must export between 10g and pre-10g databases. This article stepped through the process of performing FULL exports as
these are typical in Oracle environment. If you are doing schema or table exports the change is simple and we will visit those
in subsequent parts to this series.
On our quest to learn about Oracle's Data Pump utility it has often been compared to the old export and import (exp & imp)
utilities that we have all grown to love (or hate). This article is where where Data Pump takes a detour from these old
utilities and begins to shine. This article will explore some of the export modes available and give examples on how to export
selected object types and dependencies those objects
In order to use Data Pump, we learned in Part II of this series that a datapump directory was required to export and import
from and to databases. Here are the three setup and authorization commands needed to get started.
In the last article various FULL exports were performed. These are termed 'FULL mode' exports for the obvious reason and had
the following format.
A slight change to this example, changing the FULL keyword to SCHEMA, allows us to perform a SCHEMA mode export where a
particular schema will be exported. Anyone familiar with the old export / import (exp / imp) utilities should feel right at
home here. To export multiple schema's you need only separate each schema with commas.
C:\>expdp scott/tiger
Likewise we could change the SCHEMS option and export all objects in a particular tablespace by switching to the
TABLESPACES export mode.
C:\>expdp scott/tiger TABLESPACES=USERS DIRECTORY=datapump DUMPFILE=TSusers.dmp
LOGFILE=TSusers.log
If you wanted to export a single table, you need only switch to TABLE mode and use the following export command.
C:\>expdp scott/tiger
The interesting point to notice when issuing these commands is to take a close look at the export logs for each of these
export modes. When taking a full schema export you will notice that the export pulls out various additional object types such
as grants, roles, sequences, and views. To just name a few. Here is the log from the SCHEMA export performed above.
******************************************************************************
C:\ORADATA\DATAPUMP\SCOTT.DMP
If we then take a look at the export for a tables you will quickly notice that not all the object types that were exported for
the SCHEMA mode have been exported for the TABLE mode. Some of this is because, in our example, the DEPT table does not
have certain dependent objects and because other object types are not at all exported even though they would seem to have
a dependency. For instance indexes, triggers, and statistics will be exported under TABLE mode but a view on the DEPT table
will not. So as a caution, be careful and examine your export logs. You may not be getting everything you think is a
dependent object.
******************************************************************************
C:\ORADATA\DATAPUMP\DEPT.DMP
One way to determine the objects that will or can be exported for the different modes is to look at the three DBA views
DATABASE_EXPORT_OBJECTS, SCHEMA_EXPORT_OBJECTS, and TABLE_EXPORT_OBJECTS. Each of these views, if queried, will
give you a list and short description on the specific paths to object types that you can expect INCLUDE or EXCLUDE to be
dependent on the object you are exporting or importing. For instance if you were to query the TABLE_EXPORT_OBJECTS view
with the following SQL you would get a list of all objects that are dependent on exporting a table. As you can see there is no
entry for exporting views based on a table export. In actuality there are 86 INCLUCE/EXCLUDE types just in the
TABLE_EXPORT_OBJECTS view and many more the other two export views. I would encourage you to select the object paths
for each of the views and get acquainted with what you can export.
SQL> SELECT object_path, comments FROM table_export_objects where object_path like 'TABLE%';
OBJECT_PATH COMMENTS
------------------------------------------------------- --------------------------------------------------
TABLE/INDEX Indexes
TABLE_EXPORT/TABLE/INDEX Indexes
C:\>expdp scott/tiger
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
OGFILE=dept.log
******************************************************************************
C:\ORADATA\DATAPUMP\DEPT.DMP
When playing around with Data Pump export and using the INCLUDE / EXCLUDE feature, I soon found out that it was much
easier to use a parameter file (parfile) when specifying the different INCLUDE / EXCLUDE options. This is the same concept as
the old export and import (exp & imp) utilities. This is easier because in the course of trying to put all of the potential
options on one command line and with the fact that there are “special” characters required when specifying INCLUCE /
EXCLUDE options, you will soon find it easier to add to and subtract from the export command. I tried a number of times
putting these options on a single command line but had numerous issues. So I would suggest just getting use to the parfile
from the start.
For an example in using the parfile I decided to export the DEPT table from the SCOTT schema and include views. Remember,
as noted earlier in this article that views are not available to export under a table. So if you were to look at the DBA views,
also noted above, you need to at least go up to a schema export to include views. So I created the following parfile. This will
actually export all views in the SCOTT schema. If you knew the view names associated with the DEPT table you could also
create in IN list much like the INCLUDE statement for the DEPT table.
Parfile dept.par
SCHEMAS=SCOTT
INCLUDE=TABLE:"IN ('DEPT')"
INCLUDE=VIEW
DIRECTORY=datapump
DUMPFILE=dept.dmp
LOGFILE=dept.log
Here is the command line that would be issued. Looks very similar to the old export utility exp.
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
******************************************************************************
C:\ORADATA\DATAPUMP\DEPT.DMP
Data Pump's import command is much the same as the export command. I actually just replaced the expdp with the impdp
command in these examples and had no problems importing back into my database. Many times though we want to import
into a different schema and this is accomplished by the REMAP_SCHEMA option. Here is an example where I imported the
DEPT table into a different schema.
Oracle's Data Pump utility has many options that allow you to fine tune what you can export from a database. Just remember
to query the DBA views (DATABASE_EXPORT_OBJECTS, SCHEMA_EXPORT_OBJECTS, and TABLE_EXPORT_OBJECTS) that dictate
the dependent objects that will be exported under certain scenarios. Also keep in mind you can just as easily exclude these
object types to pull out exactly what you want.