Call Java JAR methods from ODI

Several times we will see the need of invoking custom Java programs from ODI. Uli has written an excellent article on this.
This is a good solution for the development environment.
However, the limitation of this approach lies in the fact that mandates the need to copy the Jar file to the drivers directory of ODI.
This also needs the agent to be bounced each time a new jar is added.
Think of a scenario where Java agnostic developers wish to use ODI. This is a maintenance overhead for an ODI administrator and also will clutter up the ODI drivers directory. In my opinion, the ODI drivers directory should be used only for Database drivers or application adaptors.

Any custom java code should be used in the following way:

import os
import sys

jars = [
    "/home/users/ankit/java/MyFileWrite.jar"
    ]

for jar in jars:
   sys.path.append(jar)

from com.mycompany.MyFileWrite import *

fw=MyFileWrite()
fw.writeFile()

ODI Object versioning

Versioning in ODI is a less covered topic.
Lets look today at how ODI handles object versioning. If you recall from the ODI repository architecture diagrams, ODI stores Topology, Security and Versioning information in the Master Repository.
Each time an ODI object (Project, Model, Datastore, etc) is versioned, it is saved in the Master Repository.

Eg. If we version the package - PKG_ODI_PROCESS as follows:


Then, in ODI_MASTER_REPOSITORY, we can see that there are 2 new records added to the SNP_VERSION table. The following query can be used:

SELECT i_instance, ext_version, i_data, instance_name
FROM snp_version
WHERE 1 = 1
and instance_name LIKE 'PKG_ODI_PROCESS'

In here,
I_INSTANCE is the Internal ID of the object in the work repository
EXT_VERSION is the 1.0.0.1 notation version assigned to the object
I_DATA is the FK into SNP_DATA table that stores the versioned object
INSTANCE_NAME is the name of the object itself at the time of being versioned

SNP_VERSION works alongwith SNP_DATA to store the actual versioned object. We can get that information using the following query:
SELECT sv.i_instance,
sv.i_objects,
sv.ext_version,
sv.i_data,
sv.instance_name,
sd.data_contents
FROM snp_version sv, snp_data sd
WHERE 1 = 1
and sv.i_data = sd.i_data
and instance_name LIKE 'PKG_ODI_PROCESS'

The DATA_CONTENTS column is of type LONG_RAW and stores the versioned object in a compressed LZW compression form (Unix compress command)

We can save the contents of SNP_DATA.DATA_CONTENTS into an Operating System file named as pkg_odi_process.Z, though we can name it whatever we wish with a .Z extension.
Uncompress this file using Unix uncompress or Windows Winzip, WinRar.
The file that you get by uncompressing, should be given a .xml extension.
The resulting pkg_odi_process.xml file is the same file as if we would had exported the package PKG_ODI_PROCESS using Export command in the Designer module into an XML file.

This file can be used to import back to the Work Repository.

Odi Load Multiple XML Files

Loading multiple XML files into Oracle using ODI is illustrated in this post.

The standard reverse engineering driver will parse the dtd or if the dtd is not available, it will parse the XML and generate a DTD on the fly in the same location where the XML file exists.

To load multiple XML files we will have to employ ODI variables. An ODI variable is needed to store the name of the file that needs to be loaded to Oracle.
Declare the variable in your project V_MYXMLFILE with the following query:
select xmlfilename from my_metadata_table.

This my_metadata_table should exist in an Oracle schema and should contain a column xmlfilename which lists all the XML filenames (alongwith path).
Each time the variable is refreshed, it will populate the first value from the table. You can follow the instructions found at http://odiexperts.com/?p=524 to loop through each row in the table.

Now create 2 ODI procedures:

1. Truncate XML Schema
2. Load XML Schema

1.) Truncate XML Schema will contain 1 command "Truncate"

Technology = XML and Schema = Logical XML schema that you want to work on.
The "Command on Target" will be "TRUNCATE SCHEMA MYSCHEMA".
Here MYSCHEMA is the name of the XML schema that is defined in Topology Manager using the s tag as in
jdbc:snps:xml?d=E:/XMLDIR/myXML.dtd&s=MYSCHEMA

Truncate XML Schema will wipe out all the records in the data structure that holds the XML data. It can be in memory or it can be set to a physical database schema by using &db_props=mydbparams parameter in the xml server definition. Here mydbparams is a reference to a physical file mydbparams.properties created in the $ODI_HOME/oracledi/drivers.

2.) Load XML Schema will contain 2 commands:

Show variable value - This will be created using Jython Technology with the following commands :

a = 'Filename = '+ '#V_MYXMLFILE'
raise(a)
Also, mark the Ignore Errors = Yes, for this step.

This is cheat to display the value of the ODI variable during execution in the Operator.
Also, this step is somehow needed so that the value of the variable can be accessed by the next step.
If you remove this step, the next step (which actually loads the XML schema with the contents of the file) errors out.

Load XML Schema - This will be created using XML Technology and XML logical schema.
Command on Target = LOAD FILE "#MYPROJECT_NAME.V_MYXMLFILE" ON SCHEMA MYSCHEMA INSERT_ONLY


Now you can create a package using these procedures and refresh the variable V_MYXMLFILE with location/name of the file and execute Load XML Schema.
The procedure "Load XML Schema" can also be executed in a loop to load multiple XML files (that adhere to the same DTD) into the database tables.

ODI variables from Work Repository

Ever thought if there was an SQL way of accessing the values of the variables that were evaluated or refreshed by ODI sessions ?

ODI stores the information about the variables in the following tables in a Development WR:
SNP_VAR
SNP_VAR_DATA (historical values of variable)
SNP_VAR_SCEN
SNP_VAR_SESS

The following query will extract the variable value for a particular session:

SELECT svd.var_name,
       svd.var_n AS numval,
       svd.var_d AS dateval,
       svd.var_v AS strval
FROM snp_var_data svd,
     (SELECT sp.project_code || '.' || sv.var_name AS fullvarname
      FROM snp_project sp, snp_var sv
      WHERE sv.i_project = sp.i_project) svp,
     snp_var_sess svs
WHERE     svd.var_name = svp.fullvarname
      AND svs.var_name = svd.var_name
      AND svs.sess_no = :sess_no

The Execution WR would be a bit different

LKM File to Oracle SQLLDR ODI

One of the most common problem people are facing while using Oracle Data Integrator's LKM File to Oracle (SQLLDR) is that the step which invokes the OS based sqlldr results in an error even if LOA_ERRORS is set to non-zero number.

For eg. you have a file that contains 1000 records and 10 of them are erroneous (bad), then ODI will create a .bad file and move the records in there. Theoretically, you will expect that setting up the LKM parameter LOA_ERRORS should solve the problem. So, if you want a maximum of 20 records to error out, you will set LOA_ERRORS=20.
But, even if the number of actual errors is less than 20, ODI will indicate failure of that step.

This is a known bug 8560194 with ODI LKM File to Oracle (SQLLDR) and it has not been resolved yet.
Lets get into the details as to why this is a bug. This has to do with the way sqlldr returns the code when it terminates. Following is the list of codes that sqlldr returns for different conditions:

1. SQLLDR if successful returns 0
2. SQLLDR if unsuccessful returns 1
3. SQLLDR if successful but even 1 record erred out to .bad or .dsc file returns 2

For ODI anything that is not 0 is an error. And that will be true for any system that is dependent on the return codes

To resolve this bug, you will have to customize this KM and change the step "Invoke SQLLDR" to selectively handle each error condition.
Follow the steps:

1. Duplicate the KM.
2. Goto the step Call sqlldr. and change the technology to Jython.
3. Replace the existing code with the following:

import os
retVal = os.system(r'sqlldr control=<%=snpRef.getSrcTablesList("", "[WORK_SCHEMA]/[TABLE_NAME].ctl", "", "")%> log=<%=snpRef.getSrcTablesList("", "[WORK_SCHEMA]/[TABLE_NAME].log", "", "")%> userid=<%=snpRef.getInfo("DEST_USER_NAME")%>/<%=snpRef.getInfo("DEST_PASS")%>@<%=snpRef.getInfo("DEST_DSERV_NAME")%> > <%=snpRef.getSrcTablesList("", "[WORK_SCHEMA]/[TABLE_NAME].out", "", "")%>')

if retVal == 1 or retVal > 2:
----->raise 'SQLLDR failed. Please check the <%=snpRef.getSrcTablesList("", "[WORK_SCHEMA]/[TABLE_NAME].log", "", "")%> for details '

Replace the -----> with spaces or tab. You can enhance the above code by putting variables and further checking for the .bad, .dsc and .log files.