Страницы

четверг, 9 июня 2016 г.

opatch and libpicl on Solaris SPARC

This morning while applying recent DBBP (Database Bundle Patch) the datapatch invocation finished with an error:
SQL Patching tool version 12.1.0.2.0 on Thu Jun  2 10:58:41 2016
Copyright (c) 2015, Oracle.  All rights reserved.

Log file for this invocation: /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done

Queryable inventory could not determine the current opatch status.
Execute 'select dbms_sqlpatch.verify_queryable_inventory from dual'
and/or check the invocation log
/export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log
for the complete error.
Prereq check failed, exiting without installing any patches.

Please refer to MOS Note 1609718.1 and/or the invocation log
/export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Thu Jun  2 10:58:50 2016
Here are the contents of sqlpatch_invocation.log file:
SQL Patching tool version 12.1.0.2.0 on Thu Jun  2 10:58:41 2016
Copyright (c) 2015, Oracle.  All rights reserved.

Log file for this invocation: /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log

SQL Patching arguments:
  verbose: 1
  force: 0
  prereq: 0
  upgrade_mode_only: 
  oh: 
  bundle_series: 
  ignorable_errors: 
  bootstrap: 
  skip_upgrade_check: 
  userid: 
  pdbs: 

Connecting to database...OK
catcon: ALL catcon-related output will be written to /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_catcon__catcon_12392.lst
catcon: See /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_catcon_*.log files for output generated by scripts
catcon: See /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_catcon__*.lst files for spool files, if any
Bootstrapping registry and package to current versions...done
verify_queryable_inventory returned ORA-20001: Latest xml inventory is not loaded into table

Queryable inventory could not determine the current opatch status.
Execute 'select dbms_sqlpatch.verify_queryable_inventory from dual'
and/or check the invocation log
/export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log
for the complete error.
Prereq check failed, exiting without installing any patches.

Please refer to MOS Note 1609718.1 and/or the invocation log
/export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_12392_2016_06_02_10_58_41/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Thu Jun  2 10:58:50 2016
The famous "ORA-20001: Latest xml inventory is not loaded into table" error even has a dedicated article on MOS: Queryable Patch Inventory - Issues/Solutions for ORA-20001: Latest xml inventory is not loaded into table (Doc ID 1602089.1).
I tried all of the provided solutions but no one helped me.
As the sqlpatch_invocation.log suggested, I executed the dbms_sqlpatch.verify_queryable_inventory function:
SQL>select dbms_sqlpatch.verify_queryable_inventory from dual;

VERIFY_QUERYABLE_INVENTORY
--------------------------------------------------------------------------------
ORA-20001: Latest xml inventory is not loaded into table

Well, it was nothing new. Then I queried OPATCH_XML_INV external table because it is used internally by the aforementioned function:
SQL>select * from OPATCH_XML_INV ;
ERROR:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04004: error while reading file
/export/home/velikikh/oracle/12.1.0.2/home1/QOpatch/qopiprep.bat



no rows selected

Ok, at least I got some steps for further investigation. qopiprep.bat script simply executes opatch command with some additional processing:
$ORACLE_HOME/OPatch/opatch lsinventory -xml  $ORACLE_HOME/QOpatch/xml_file.xml -retry 0 -invPtrLoc $ORACLE_HOME/oraInst.loc >> $ORACLE_HOME/QOpatch/stout.txt
I decided to try executing it manually:
SQL>!opatch lsinv
Java HotSpot(TM) 64-Bit Server VM warning: PICL (libpicl.so.1) is missing. Performance will not be optimal.
Java HotSpot(TM) 64-Bit Server VM warning: PICL (libpicl.so.1) is missing. Performance will not be optimal.
Oracle Interim Patch Installer version 12.1.0.1.12
Copyright (c) 2016, Oracle Corporation.  All rights reserved.
..skipped...
These lines with JVM warning were rather unusual for me and I did not know why they occured.
As I had a feeling that they were related to my problem, I searched through the MOS knowledge base and found an article: VM Warning "PICL (libpicl.so.1) is missing" After Upgrading to Java SE 8u40+ or 7u80+ (Doc ID 2093623.1).
That had a reference to Poonam Bajaj's blog:
 Please note that this warning just indicates that the JVM could not load the PICL library so it won't be able to turn on some of the JIT Compiler optimizations that require the cache line data size information. 
It does not stop the Java process, or impact the process in any other way. 
Also, note that this is not a Java bug. The warning message is just intended to inform the user that the platform it is running on does not have the PICL library.
Well, nothing critical was in those messages but Queryable Opatch Inventory (hereinafter QOpatch) was not working.
To verify my assumption about the negative effect of libpicl message on QOpatch, I made a quick and dirty correction to qopiprep.bat file - I redirected the stderr to /dev/null :):
$ORACLE_HOME/OPatch/opatch lsinventory -xml  $ORACLE_HOME/QOpatch/xml_file.xml -retry 0 -invPtrLoc $ORACLE_HOME/oraInst.loc >> $ORACLE_HOME/QOpatch/stout.txt 2>/dev/null
Having done that, I executed select against OPATCH_XML_INV table:
SQL>select * from opatch_xml_inv;

XML_INVENTORY
--------------------------------------------------------------------------------
  <ora
Allright, at least it started working. I didn't try executing other commands but instead asked the SA (System Administrator) team to install the libpicl.so library.
They installed pkg:/system/libpicl and I finally got datapatch working:
[velikikh@gsmcat OPatch]$ ./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 on Thu Jun  2 13:26:15 2016
Copyright (c) 2015, Oracle.  All rights reserved.

Log file for this invocation: /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/sqlpatch_6653_2016_06_02_13_26_15/sqlpatch_invocation.log

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of SQL patches:
Bundle series DBBP:
  ID 160419 in the binary registry and not installed in the SQL registry

Adding patches to installation queue and performing prereq checks...
Installation queue:
  Nothing to roll back
  The following patches will be applied:
    22806133 (DATABASE BUNDLE PATCH: 12.1.0.2.160419 (22806133))

Installing patches...
Patch installation complete.  Total patches installed: 1

Validating logfiles...
Patch 22806133 apply: SUCCESS
  logfile: /export/home/velikikh/oracle/cfgtoollogs/sqlpatch/22806133/20067756/22806133_apply_GSMCAT_2016Jun02_13_28_31.log (no errors)
SQL Patching tool complete on Thu Jun  2 13:32:19 2016
So the conclusion is, while using Oracle Database 12c on Solaris 11 SPARC64, you will definitely need to install the libpicl. This requirement has not mentioned either in Oracle Database documentation or MOS articles (at least I have missed something) but it is present.
From Oracle Database Installation Guide for Oracle Solaris:
The following packages must be installed:
    pkg://solaris/system/dtrace
    pkg://solaris/developer/assembler
    pkg://solaris/developer/build/make
    pkg://solaris/system/xopen/xcu4 (if not already installed as part of standard Oracle Solaris 11 installation)
    pkg://solaris/x11/diagnostic/x11-info-clients
    pkg://solaris/compress/unzip
Although I'd had these packages installed, the datapatch execution completed with the errors till I installed system/picl:
-bash-4.1$ pkg list dtrace assembler make xcu4 x11-info-clients unzip
NAME (PUBLISHER)                                  VERSION                    IFO
compress/unzip                                    6.0-0.175.2.10.0.4.0       i--
developer/assembler                               0.5.11-0.175.2.13.0.2.0    i--
developer/build/make                              0.5.11-0.175.2.0.0.34.0    i--
system/dtrace                                     0.5.11-0.175.2.12.0.4.2    i--
system/xopen/xcu4                                 0.5.11-0.175.2.13.0.1.2    i--
x11/diagnostic/x11-info-clients                   7.7-0.175.2.0.0.42.1406    i--
-bash-4.1$ pkg list picl
NAME (PUBLISHER)                                  VERSION                    IFO
system/picl                                       0.5.11-0.175.2.9.0.3.2     i--

суббота, 14 мая 2016 г.

Asynchronous Global Index Maintenance effects

I think Asynchronous Global Index Maintenance is one of the most exciting features in Oracle Database 12c.
I would like to blog about one particular case when that feature can cause performance problems.
Let's create some test data:
SQL> create table t
  2  partition by range(owner)
  3  (
  4    partition values less than ('SYS'),
  5    partition values less than (maxvalue)
  6  )
  7  as
  8  select *
  9    from dba_objects;
SQL> 
SQL> create index t_name_i on t(object_name);
Consider the following query:
SQL> explain plan for
  2  select max(object_name)
  3    from t;
SQL> select *
  2    from table(dbms_xplan.display);


PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------
Plan hash value: 2886567490

---------------------------------------------------------------------------------------
| Id  | Operation                  | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT           |          |     1 |    25 |     3   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE            |          |     1 |    25 |            |          |
|   2 |   INDEX FULL SCAN (MIN/MAX)| T_NAME_I |     1 |    25 |     3   (0)| 00:00:01 |
---------------------------------------------------------------------------------------
It's quite simple and the queries like that are common in my environment. For instance, they are used to monitor application activities: "select max(date_column) from some_table" and the stuff alike.
Now I'm going to drop a partition of that table:
SQL> alter table t drop partition for ('X') update indexes;
The index is marked as having orphaned entries after that:
SQL> select index_name, orphaned_entries
  2    from ind
  3   where table_name='T';


INDEX_NAME                     ORPHANED_
------------------------------ ---------
T_NAME_I                       YES
Causing some important changes in the plan:
SQL> explain plan for
  2  select max(object_name)
  3    from t;
SQL> select *
  2    from table(dbms_xplan.display);


PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------
Plan hash value: 3249307143

----------------------------------------------------------------------------------
| Id  | Operation             | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------
|   0 | SELECT STATEMENT      |          |     1 |    25 |   148   (1)| 00:00:01 |
|   1 |  SORT AGGREGATE       |          |     1 |    25 |            |          |
|*  2 |   INDEX FAST FULL SCAN| T_NAME_I | 96566 |  2357K|   148   (1)| 00:00:01 |
----------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter(TBL$OR$IDX$PART$NUM("T",0,8,0,"T".ROWID)=1)
The filter clearly was occured due to orphaned index entries which is confirmed by looking into the CBO traces (event 10053):
Added Filter for Orphaned Entries of Index T_NAME_I:
TBL$OR$IDX$PART$NUM("T",0,8,0,"T".ROWID)=1
The IFFS (INDEX FAST FULL SCAN) would cause a huge amount of I/O being executed against large indexes.
For instance, some queries started scanning multi-gigabyte indexes in my database after I had dropped a single small partition while executing periodic maintenance tasks.
Of course, in such case Global Index Maintenance can't be delayed and an index synchronization procedure must be executed ASAP:
SQL> alter index t_name_i coalesce cleanup;
SQL> select index_name, orphaned_entries
  2    from ind
  3   where table_name='T';


INDEX_NAME                     ORPHANED_
------------------------------ ---------
T_NAME_I                       NO
SQL> explain plan for
  2  select max(object_name)
  3    from t;
SQL> select * from table(dbms_xplan.display);


PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------
Plan hash value: 2886567490

---------------------------------------------------------------------------------------
| Id  | Operation                  | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT           |          |     1 |    25 |     3   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE            |          |     1 |    25 |            |          |
|   2 |   INDEX FULL SCAN (MIN/MAX)| T_NAME_I |     1 |    25 |     3   (0)| 00:00:01 |
---------------------------------------------------------------------------------------

среда, 30 декабря 2015 г.

ORA-8102 on ALTER TABLE SHRINK SPACE with default column optimization

We have been using centralized AWR repository since 2013.
I developed some PL/SQL procedures for that using original awrextr.sql/awrload.sql scripts as a template.
That solution have some drawbacks, for example, an AWR retention for foreign databases always set to 40150 years after each load:
SQL> select dbid, retention from dba_hist_wr_control;

      DBID RETENTION
---------- --------------------
1110059808 +00366 00:00:00.0
1917063347 +40150 00:00:00.0
1996649024 +40150 00:00:00.0
We could change the retention but that would always be reset on the next AWR load.
To prevent the excessive AWR data growth, I executed DBMS_WORKLOAD_REPOSITORY.DROP_SNAPSHOT_RANGE procedure once a year.
The 2015 year was no exception.
I also executed ALTER TABLE SHRINK SPACE command after I had executed DBMS_WORKLOAD_REPOSITORY.DROP_SNAPSHOT_RANGE procedures.
Usually that command takes a while to execute, because the some of AWR tables are more than 50G in size, for example, WRH$_ACTIVE_SESSION_HISTORY.
It's a base table for DBA_HIST_ACTIVE_SESS_HISTORY view and contains an ASH data that's huge.
This year things went wrong:
SQL> alter table SYS.WRH$_ACTIVE_SESSION_HISTORY shrink space;
-- AFTER 4-8 hours
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 635
Session ID: 281 Serial number: 54363
alert.log was not very helpful:
Mon Dec 21 14:37:31 2015
Process 0x3cd728e40 appears to be hung while dumping
Current time = 1689592362, process death time = 1689530965 interval = 60000
Called from location UNKNOWN:UNKNOWN
Attempting to kill process 0x3cd728e40 with OS pid = 17740
OSD kill succeeded for process 3cd728e40
Mon Dec 21 14:39:46 2015
Errors in file /oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_25447.trc:
But a relevant trace file (which was more than 4G in size) spotted a light on a problem:
*** ACTION NAME:() 2015-12-21 14:39:46.890
 
oer 8102.2 - obj# 1021709, rdba: 0x086ad43c(afn 33, blk# 2806844)
kdk key 8102.2:
  ncol: 6, len: 26
  key: (26): 
 06 c5 0c 0b 06 63 09 04 c3 08 0e 3e 02 c1 02 05 c4 62 46 05 3b 03 c2 02 3d
 ff
  mask: (4096): 
 81 90 20 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Well, string "oer 8102.2" it's a sign of famous ORA-8102 error:
[oracle@localhost ~]$ oerr ora 8102
08102, 00000, "index key not found, obj# %s, file %s, block %s (%s)"
// *Cause:  Internal error: possible inconsistency in index
// *Action:  Send trace file to your customer support representative, along
//      with information on reproducing the error
obj# 1021709 in the trace file easily shows to us where the problem is:
SQL> select xmltype(cursor(select * from dba_objects where object_id=1021709)) xml_rec from dual;

XML_REC
----------------------------------------------------------------------------------------------------------


  
    SYS
    WRH$_ACTIVE_SESSION_HISTORY_PK
    WRH$_ACTIVE_1110059808_66853
    1021709
    1080082
    INDEX PARTITION
    27.10.2015 01:28:05
    22.12.2015 09:11:22
    2015-10-27:01:28:05
    VALID
    N
    N
    N
    4
    NONE
    Y
  

The index WRH$_ACTIVE_SESSION_HISTORY_PK is local partitioned on next columns:
CREATE UNIQUE INDEX "SYS"."WRH$_ACTIVE_SESSION_HISTORY_PK" ON "SYS"."WRH$_ACTIVE_SESSION_HISTORY" ("DBID", "SNAP_ID", "INSTANCE_NUMBER", "SAMPLE_ID", "SESSION_ID", "CON_DBID")
The all of the above columns are numbers.
The index key:
  key: (26): 
 06 c5 0c 0b 06 63 09 04 c3 08 0e 3e 02 c1 02 05 c4 62 46 05 3b 03 c2 02 3d
 ff
can be decrypted easily:
SQL> with input as (
  2    -- index key
  3    select '06 c5 0c 0b 06 63 09 04 c3 08 0e 3e 02 c1 02 05 c4 62 46 05 3b 03 c2 02 3d ff' c from dual),
  4    t(pos, num,c) as (
  5    -- traverse the index key recursively
  6    select 0 pos, 0, replace(c, ' ') c from input union all
  7    select pos+1,
  8           utl_raw.cast_to_number(hextoraw(substr(c, 3, to_number(substr(c,1,2), 'xx')*2))),
  9           substr(c, (to_number(substr(c, 1, 2), 'xx')+1)*2+1)
 10      from t
 11     where c<>'ff')
 12  select pos, num
 13    from t
 14   where pos>0
 15   order by pos
 16  /

       POS        NUM
---------- ----------
         1 1110059808
         2      71361
         3          1
         4   97690458
         5        160
Ok, now we've known that: dbid=1110059808, snap_id=71361, instance_number=1, sample_id=97690458, session_id=160.
But the index was built on six columns when the problem key contains only five.
What is the CON_DBID for the problem key? That's where the trouble lies!
CON_DBID was introduced in Oracle 12.1 and added as a "DEFAULT NOT NULL" column, i.e. that column addition was a metadata-only operation without updating the table blocks (unless _add_col_optim_enabled was not set to FALSE).
SQL> select to_char(property, 'fm0xxxxxxx')
  2    from sys.col$
  3   where obj# = 22800
  4     and name = 'CON_DBID';

TO_CHAR(PROPERTY,'FM0XXXXXX
---------------------------
40000000
Notice that property was 0x40000000 - I speculated that such property was set to columns which were added with default column optimization.
Some MOS notes prove that, i.e. Table Columns Have Wrong Default Value After Transporting Tablespaces (Doc ID 1602994.1).
I dumped the relevant table block and found that CON_DBID column was not present in it.
The pieces of that puzzle started to fit together.
I had constructed a simple test case which reproduced an ORA-8102 error:
SQL> create table t(x int, pad varchar2(100)) enable row movement;
SQL> insert /*+ append*/
  2    into t
  3  select level, lpad('x', 100, 'x')
  4    from dual
  5    connect by level<=1e5;
SQL> commit;
SQL> 
SQL> alter table t add y int default 10 not null;
SQL> 
SQL> create index t_xy_i on t(x,y);
SQL> 
SQL> delete t where x<=1e5/2;
SQL> commit;
SQL> 
SQL> alter table t shrink space;
alter table t shrink space
*
ERROR at line 1:
ORA-08102: index key not found, obj# 91957, file 10, block 3990 (2)
The problem is present in Oracle 11.2.0.4 and more newest versions. I didn't test in earlier 11g patchsets.
Oracle Support raised a new bug: ORA-8102 ON ALTER TABLE SHRINK SPACE WITH ADD COL OPTIMIZATION (unpublished), which is still under investigation.
The tables suspected to an ORA-8102 error can be identified easily by bitand(col$.property, 1073741824)=1073741824 (0x40000000 in hex):
SQL> select o.object_name
  2    from sys.col$ c,
  3         dba_objects o
  4   where bitand(c.property, 1073741824)=1073741824
  5     and o.object_id=c.obj#
  6     and o.owner='SYS'
  7   order by o.object_name;
I restricted above query to SYS schema. Below is the output of that query in one of database which was upgraded from 11.2.0.4 to 12.1.0.2 in the 2015:
SQL> select o.object_name
  2    from sys.col$ c,
  3         dba_objects o
  4   where bitand(c.property, 1073741824)=1073741824
  5     and o.object_id=c.obj#
  6     and o.owner='SYS'
  7   order by o.object_name;

OBJECT_NAME
-------------------------------------------------------------------
CDB_LOCAL_ADMINAUTH$
HISTGRM$
PROFNAME$
WRH$_ACTIVE_SESSION_HISTORY
WRH$_BG_EVENT_SUMMARY
WRH$_BUFFERED_QUEUES
WRH$_BUFFERED_SUBSCRIBERS
WRH$_BUFFER_POOL_STATISTICS
WRH$_CLUSTER_INTERCON
WRH$_COMP_IOSTAT
WRH$_CR_BLOCK_SERVER
WRH$_CURRENT_BLOCK_SERVER
WRH$_DATAFILE
WRH$_DB_CACHE_ADVICE
WRH$_DISPATCHER
WRH$_DLM_MISC
WRH$_DYN_REMASTER_STATS
WRH$_DYN_REMASTER_STATS
WRH$_ENQUEUE_STAT
WRH$_EVENT_HISTOGRAM
WRH$_EVENT_NAME
WRH$_FILEMETRIC_HISTORY
WRH$_FILESTATXS
WRH$_IC_CLIENT_STATS
WRH$_IC_DEVICE_STATS
WRH$_INSTANCE_RECOVERY
WRH$_INST_CACHE_TRANSFER
WRH$_INTERCONNECT_PINGS
WRH$_IOSTAT_DETAIL
WRH$_IOSTAT_FILETYPE
WRH$_IOSTAT_FILETYPE_NAME
WRH$_IOSTAT_FUNCTION
WRH$_IOSTAT_FUNCTION_NAME
WRH$_JAVA_POOL_ADVICE
WRH$_LATCH
WRH$_LATCH_CHILDREN
WRH$_LATCH_MISSES_SUMMARY
WRH$_LATCH_NAME
WRH$_LATCH_PARENT
WRH$_LIBRARYCACHE
WRH$_LOG
WRH$_MEMORY_RESIZE_OPS
WRH$_MEMORY_TARGET_ADVICE
WRH$_MEM_DYNAMIC_COMP
WRH$_METRIC_NAME
WRH$_MTTR_TARGET_ADVICE
WRH$_MUTEX_SLEEP
WRH$_MVPARAMETER
WRH$_OPTIMIZER_ENV
WRH$_OSSTAT
WRH$_OSSTAT_NAME
WRH$_PARAMETER
WRH$_PARAMETER_NAME
WRH$_PERSISTENT_QMN_CACHE
WRH$_PERSISTENT_QUEUES
WRH$_PERSISTENT_SUBSCRIBERS
WRH$_PGASTAT
WRH$_PGA_TARGET_ADVICE
WRH$_PLAN_OPERATION_NAME
WRH$_PLAN_OPTION_NAME
WRH$_PROCESS_MEMORY_SUMMARY
WRH$_RESOURCE_LIMIT
WRH$_ROWCACHE_SUMMARY
WRH$_RSRC_CONSUMER_GROUP
WRH$_RSRC_PLAN
WRH$_RULE_SET
WRH$_SEG_STAT
WRH$_SEG_STAT_OBJ
WRH$_SERVICE_NAME
WRH$_SERVICE_STAT
WRH$_SERVICE_WAIT_CLASS
WRH$_SESSMETRIC_HISTORY
WRH$_SESS_TIME_STATS
WRH$_SESS_TIME_STATS
WRH$_SGA
WRH$_SGASTAT
WRH$_SGA_TARGET_ADVICE
WRH$_SHARED_POOL_ADVICE
WRH$_SHARED_SERVER_SUMMARY
WRH$_SQLCOMMAND_NAME
WRH$_SQLSTAT
WRH$_SQLTEXT
WRH$_SQL_BIND_METADATA
WRH$_SQL_PLAN
WRH$_SQL_SUMMARY
WRH$_SQL_WORKAREA_HISTOGRAM
WRH$_STAT_NAME
WRH$_STREAMS_APPLY_SUM
WRH$_STREAMS_APPLY_SUM
WRH$_STREAMS_CAPTURE
WRH$_STREAMS_CAPTURE
WRH$_STREAMS_POOL_ADVICE
WRH$_SYSMETRIC_HISTORY
WRH$_SYSMETRIC_SUMMARY
WRH$_SYSSTAT
WRH$_SYSTEM_EVENT
WRH$_SYS_TIME_MODEL
WRH$_TABLESPACE
WRH$_TABLESPACE_SPACE_USAGE
WRH$_TABLESPACE_STAT
WRH$_TEMPFILE
WRH$_TEMPSTATXS
WRH$_THREAD
WRH$_TOPLEVELCALL_NAME
WRH$_UNDOSTAT
WRH$_WAITCLASSMETRIC_HISTORY
WRH$_WAITSTAT
WRI$_ADV_SQLT_PLAN_HASH
WRI$_OPTSTAT_HISTGRM_HISTORY
WRI$_SQLSET_BINDS
WRI$_SQLSET_DEFINITIONS
WRI$_SQLSET_MASK
WRI$_SQLSET_PLANS
WRI$_SQLSET_PLAN_LINES
WRI$_SQLSET_STATEMENTS
WRI$_SQLSET_STATISTICS
WRI$_SQLTEXT_REFCOUNT

117 rows selected.
You could see that mostly AWR tables are affected (WRH$%, WRI%). The brand new 12c database's AWR tables won't have property set to 1073741824 (0x40000000), unless you wouldn't create some.
If you set the _add_col_optim_enabled parameter to FALSE, you will be safe.
If you will face a same issue one day, the possible workaround for the issue is to rebuild table in anyway to update the problem column (MOVE table, redefine table online, or set column value equals to self).

понедельник, 30 ноября 2015 г.

V$SQL_PLAN_MONITOR.STARTS is higher than expected due to NLJ batching/Prefetching

Recently one of the developers asked me to explain why V$SQL_PLAN_MONITOR.STARTS is higher than expected for one particular query.
Here is a problem query (some columns are hidden to preserve readability):
SQL Plan Monitoring Details (Plan Hash Value=40624586)
================================================================================================================================================
| Id |                Operation                |        Name        | Execs |   Rows   | Read  | Read  | Activity |      Activity Detail       |
|    |                                         |                    |       | (Actual) | Reqs  | Bytes |   (%)    |        (# samples)         |
================================================================================================================================================
|  0 | SELECT STATEMENT                        |                    |     1 |        0 |       |       |          |                            |
|  1 |   NESTED LOOPS                          |                    |     1 |        0 |       |       |          |                            |
|  2 |    NESTED LOOPS                         |                    |     1 |    62594 |       |       |          |                            |
|  3 |     TABLE ACCESS BY INDEX ROWID BATCHED | MAIN_TABLE         |     1 |    62594 | 22777 | 178MB |    25.89 | Cpu (1)                    |
|    |                                         |                    |       |          |       |       |          | db file parallel read (28) |
|  4 |      INDEX RANGE SCAN                   | MAIN_TABLE_I       |     1 |    62594 |    25 | 200KB |          |                            |
|  5 |     INDEX UNIQUE SCAN                   | CHILD_TABLE_PK     |  109K |    62594 | 63798 | 498MB |    71.43 | Cpu (2)                    |
|    |                                         |                    |       |          |       |       |          | db file parallel read (78) |
|  6 |    TABLE ACCESS BY INDEX ROWID          | CHILD_TABLE        |  105K |        0 |  1402 |  11MB |     1.79 | db file parallel read (2)  |
================================================================================================================================================
You can see that we've obtained 62K rows at step 3, but notice a number of "Execs" at step 5: 109K. It's almost twice as higher than 62K.
Number of rows ("Rows (Actual)") is correct, though.
I checked relevant columns in V$SQL_PLAN_MONITOR view (STARTS/OUTPUT_ROWS) and verified that there's no contradiction between V$SQL_PLAN_MONITOR view and DBMS_SQLTUNE.REPORT_SQL_MONITOR output.
I investigated this issue further and constructed a simple test case which can be used to reproduce the issue.
SQL> create table fact
  2  as
  3  select date '2015-08-01' + trunc(level/4)/86400 fact_date,
  4         lpad('x', 240, 'x') padding,
  5         mod(level, 100000) dim_id
  6    from dual
  7    connect by level<=4*86400;

Table created.

SQL> 
SQL> exec dbms_stats.gather_table_stats( '', 'fact')

PL/SQL procedure successfully completed.

SQL> 
SQL> create table dim
  2  as
  3  select trunc(dbms_random.value(1,100000)) id,
  4         lpad('x', 340, 'x') padding
  5    from dual
  6    connect by level<=2*86400;

Table created.

SQL> 
SQL> create index dim_i on dim(id);

Index created.

SQL> 
SQL> exec dbms_stats.gather_table_stats( '', 'dim')

PL/SQL procedure successfully completed.
I created 2 tables: FACT and DIM, populated them with data and gathered statistics.
Now, I will flush the buffer cache and execute a test query:
SQL> alter system flush buffer_cache;

System altered.

SQL> 
SQL> select /*+ monitor leading(f) use_nl(d) full(f)*/
  2         count(f.padding),
  3         count(d.padding)
  4    from fact f,
  5         dim d
  6   where f.fact_date between to_date('01.08.2015 12:00', 'dd.mm.yyyy hh24:mi') and to_date('01.08.2015 12:10', 'dd.mm.yyyy hh24:mi')
  7     and d.id = f.dim_id;


COUNT(F.PADDING) COUNT(D.PADDING)
---------------- ----------------
            4214             4214

1 row selected.

SQL> 
Let's see the DBMS_SQLTUNE report for the last query:
SQL> select dbms_sqltune.report_sql_monitor from dual;


REPORT_SQL_MONITOR
-----------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Monitoring Report

SQL Text
------------------------------
select /*+ monitor leading(f) use_nl(d) full(f)*/ count(f.padding), count(d.padding) from fact f, dim d where f.fact_date between to_date('01.08.2015 12:00

.. skip..

SQL Plan Monitoring Details (Plan Hash Value=85884857)
===========================================================================================================================================================
| Id |            Operation            | Name  |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read | Read  | Activity |   Activity Detail    |
|    |                                 |       | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs | Bytes |   (%)    |     (# samples)      |
===========================================================================================================================================================
|  0 | SELECT STATEMENT                |       |         |       |        10 |    +14 |     1 |        1 |      |       |          |                      |
|  1 |   SORT AGGREGATE                |       |       1 |       |        10 |    +14 |     1 |        1 |      |       |          |                      |
|  2 |    NESTED LOOPS                 |       |         |       |        10 |    +14 |     1 |     4214 |      |       |          |                      |
|  3 |     NESTED LOOPS                |       |    5025 | 11890 |        10 |    +14 |     1 |     4214 |      |       |          |                      |
|  4 |      TABLE ACCESS FULL          | FACT  |    2408 |  3535 |        23 |     +1 |     1 |     2404 |  109 | 100MB |     8.70 | Cpu (1)              |
|    |                                 |       |         |       |           |        |       |          |      |       |          | direct path read (1) |
|  5 |      INDEX RANGE SCAN           | DIM_I |       2 |     1 |        10 |    +14 |  2539 |     4214 |  562 |   4MB |          |                      |
|  6 |     TABLE ACCESS BY INDEX ROWID | DIM   |       2 |     4 |        22 |     +2 |  7582 |     4214 | 2810 |  22MB |    91.30 | Cpu (21)             |
===========================================================================================================================================================
Ok, we've just reproduced the issue: row source 4 of the plan generated 2404 rows, but the number of "Execs" (2539) at line 5 is equals to 2539, which is slightly greater.
It was not by accident that I flushed the buffer cache before I execute a query.
If I'll execute a query once again, then dbms_sqltune.report_sql_monitor report has number of "Execs" in line 5 equals to number of rows in line 4:
SQL Plan Monitoring Details (Plan Hash Value=85884857)
======================================================================================================================================================
| Id |            Operation            | Name  |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read | Read  | Activity | Activity Detail |
|    |                                 |       | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs | Bytes |   (%)    |   (# samples)   |
======================================================================================================================================================
|  0 | SELECT STATEMENT                |       |         |       |         1 |     +0 |     1 |        1 |      |       |          |                 |
|  1 |   SORT AGGREGATE                |       |       1 |       |         1 |     +0 |     1 |        1 |      |       |          |                 |
|  2 |    NESTED LOOPS                 |       |         |       |         1 |     +0 |     1 |     4214 |      |       |          |                 |
|  3 |     NESTED LOOPS                |       |    5025 | 11890 |         1 |     +0 |     1 |     4214 |      |       |          |                 |
|  4 |      TABLE ACCESS FULL          | FACT  |    2408 |  3535 |         1 |     +0 |     1 |     2404 |  108 | 100MB |          |                 |
|  5 |      INDEX RANGE SCAN           | DIM_I |       2 |     1 |         1 |     +0 |  2404 |     4214 |      |       |          |                 |
|  6 |     TABLE ACCESS BY INDEX ROWID | DIM   |       2 |     4 |         1 |     +0 |  4214 |     4214 |      |       |          |                 |
======================================================================================================================================================
I believe, that differences in execs/rows were caused by NLJ batching/Table prefetching.
How can we prove it?
First, I'll show how changes "physical reads cache prefetch" before/after query execution.
SQL> select n.name, s.value
  2    from v$statname n, v$mystat s
  3   where n.name like '%prefetch%'
  4     and s.statistic#=n.statistic#
  5     and s.value>0
  6   order by n.name;


NAME                                VALUE
------------------------------ ----------
physical reads cache prefetch      127987

1 row selected.

SQL> 
SQL> select /*+ monitor leading(f) use_nl(d) full(f)*/
  2         count(f.padding),
  3         count(d.padding)
  4    from fact f,
  5         dim d
  6   where f.fact_date between to_date('01.08.2015 12:00', 'dd.mm.yyyy hh24:mi') and to_date('01.08.2015 12:10', 'dd.mm.yyyy hh24:mi')
  7     and d.id = f.dim_id;


COUNT(F.PADDING) COUNT(D.PADDING)
---------------- ----------------
            4214             4214

1 row selected.

SQL> 
SQL> select n.name, s.value
  2    from v$statname n, v$mystat s
  3   where n.name like '%prefetch%'
  4     and s.statistic#=n.statistic#
  5     and s.value>0
  6   order by n.name;


NAME                                VALUE
------------------------------ ----------
physical reads cache prefetch      131321

1 row selected.
Statistic increased by the 131321-127987=3334 blocks. Second, when I disable nlj_batching, there are no differences in execs/rows:
SQL> alter system flush buffer_cache;

System altered.

SQL> 
SQL> select n.name, s.value
  2    from v$statname n, v$mystat s
  3   where n.name like '%prefetch%'
  4     and s.statistic#=n.statistic#
  5     and s.value>0
  6   order by n.name;


NAME                                VALUE
------------------------------ ----------
physical reads cache prefetch      137989

1 row selected.

SQL> 
SQL> select /*+ monitor leading(f) use_nl(d) full(f) opt_param('_nlj_batching_enabled' 0)*/
  2         count(f.padding),
  3         count(d.padding)
  4    from fact f,
  5         dim d
  6   where f.fact_date between to_date('01.08.2015 12:00', 'dd.mm.yyyy hh24:mi') and to_date('01.08.2015 12:10', 'dd.mm.yyyy hh24:mi')
  7     and d.id = f.dim_id;


COUNT(F.PADDING) COUNT(D.PADDING)
---------------- ----------------
            4214             4214

1 row selected.

SQL> 
SQL> select n.name, s.value
  2    from v$statname n, v$mystat s
  3   where n.name like '%prefetch%'
  4     and s.statistic#=n.statistic#
  5     and s.value>0
  6   order by n.name;


NAME                                VALUE
------------------------------ ----------
physical reads cache prefetch      137989

1 row selected.

SQL> 
SQL> select dbms_sqltune.report_sql_monitor from dual;


REPORT_SQL_MONITOR
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Monitoring Report

SQL Text
------------------------------
select /*+ monitor leading(f) use_nl(d) full(f) opt_param('_nlj_batching_enabled' 0)*/ count(f.padding), count(d.padding) from fact f, dim d where f.fact_date be
 and d.id = f.dim_id

.. skip ..

SQL Plan Monitoring Details (Plan Hash Value=1381503666)
=================================================================================================================================================================
| Id |           Operation            | Name  |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read | Read  | Activity |       Activity Detail       |
|    |                                |       | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs | Bytes |   (%)    |         (# samples)         |
=================================================================================================================================================================
|  0 | SELECT STATEMENT               |       |         |       |         1 |     +2 |     1 |        1 |      |       |          |                             |
|  1 |   SORT AGGREGATE               |       |       1 |       |         1 |     +2 |     1 |        1 |      |       |          |                             |
|  2 |    TABLE ACCESS BY INDEX ROWID | DIM   |       2 |     4 |         2 |     +1 |     1 |     4214 | 3360 |  26MB |   100.00 | db file sequential read (2) |
|  3 |     NESTED LOOPS               |       |    5025 | 11890 |         1 |     +2 |     1 |     6619 |      |       |          |                             |
|  4 |      TABLE ACCESS FULL         | FACT  |    2408 |  3535 |         1 |     +2 |     1 |     2404 |  109 | 100MB |          |                             |
|  5 |      INDEX RANGE SCAN          | DIM_I |       2 |     1 |         1 |     +2 |  2404 |     4214 |   12 | 98304 |          |                             |
=================================================================================================================================================================


1 row selected.
Or, if we want to obtain "more-classic" NLJ plan shape (I added no_nlj_prefetch(d) hint):
SQL> alter system flush buffer_cache;

System altered.

SQL> 
SQL> select n.name, s.value
  2    from v$statname n, v$mystat s
  3   where n.name like '%prefetch%'
  4     and s.statistic#=n.statistic#
  5     and s.value>0
  6   order by n.name;


NAME                                VALUE
------------------------------ ----------
physical reads cache prefetch      137989

1 row selected.

SQL> 
SQL> select /*+ monitor leading(f) use_nl(d) full(f) opt_param('_nlj_batching_enabled' 0) no_nlj_prefetch(d)*/
  2         count(f.padding),
  3         count(d.padding)
  4    from fact f,
  5         dim d
  6   where f.fact_date between to_date('01.08.2015 12:00', 'dd.mm.yyyy hh24:mi') and to_date('01.08.2015 12:10', 'dd.mm.yyyy hh24:mi')
  7     and d.id = f.dim_id;


COUNT(F.PADDING) COUNT(D.PADDING)
---------------- ----------------
            4214             4214

1 row selected.

SQL> 
SQL> select n.name, s.value
  2    from v$statname n, v$mystat s
  3   where n.name like '%prefetch%'
  4     and s.statistic#=n.statistic#
  5     and s.value>0
  6   order by n.name;


NAME                                VALUE
------------------------------ ----------
physical reads cache prefetch      137989

1 row selected.

SQL> 
SQL> select dbms_sqltune.report_sql_monitor from dual;


REPORT_SQL_MONITOR
------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Monitoring Report

SQL Text
------------------------------
select /*+ monitor leading(f) use_nl(d) full(f) opt_param('_nlj_batching_enabled' 0) no_nlj_prefetch(d)*/ count(f.padding), count(d.padding) from fact f, dim d wh
d.mm.yyyy hh24:mi') and d.id = f.dim_id

.. skip ..

SQL Plan Monitoring Details (Plan Hash Value=676372893)
==================================================================================================================================================================
| Id |            Operation            | Name  |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read | Read  | Activity |       Activity Detail       |
|    |                                 |       | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs | Bytes |   (%)    |         (# samples)         |
==================================================================================================================================================================
|  0 | SELECT STATEMENT                |       |         |       |         1 |     +2 |     1 |        1 |      |       |          |                             |
|  1 |   SORT AGGREGATE                |       |       1 |       |         1 |     +2 |     1 |        1 |      |       |          |                             |
|  2 |    NESTED LOOPS                 |       |    5025 | 11890 |         1 |     +2 |     1 |     4214 |      |       |          |                             |
|  3 |     TABLE ACCESS FULL           | FACT  |    2408 |  3535 |         1 |     +2 |     1 |     2404 |  109 | 100MB |          |                             |
|  4 |     TABLE ACCESS BY INDEX ROWID | DIM   |       2 |     4 |         2 |     +2 |  2404 |     4214 | 3360 |  26MB |    66.67 | Cpu (1)                     |
|    |                                 |       |         |       |           |        |       |          |      |       |          | db file sequential read (1) |
|  5 |      INDEX RANGE SCAN           | DIM_I |       2 |     1 |         1 |     +2 |  2404 |     4214 |   12 | 98304 |          |                             |
==================================================================================================================================================================
I have searched for a similar issues through MOS and found a Bug 13634445 : V$SQL_PLAN_MONITOR AND NL BATCHING IN 11G, which has status 92 "Closed, Not a Bug".
Another interesting observation is that by adding a "gather_plan_statistics" hint, I'll disable prefetching for that case and fix V$SQL_PLAN_MONITOR "discrepancy"
SQL> alter system flush buffer_cache;

System altered.

SQL> 
SQL> select /*+ monitor leading(f) use_nl(d) full(f) gather_plan_statistics*/
  2         count(f.padding),
  3         count(d.padding)
  4    from fact f,
  5         dim d
  6   where f.fact_date between to_date('01.08.2015 12:00', 'dd.mm.yyyy hh24:mi') and to_date('01.08.2015 12:10', 'dd.mm.yyyy hh24:mi')
  7     and d.id = f.dim_id;


COUNT(F.PADDING) COUNT(D.PADDING)
---------------- ----------------
            4214             4214

1 row selected.

SQL> 
SQL> select dbms_sqltune.report_sql_monitor from dual;


REPORT_SQL_MONITOR
------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL Monitoring Report

SQL Text
------------------------------
select /*+ monitor leading(f) use_nl(d) full(f) gather_plan_statistics*/ count(f.padding), count(d.padding) from fact f, dim d where f.fact_date between to_date('
dim_id

..skip..

SQL Plan Monitoring Details (Plan Hash Value=85884857)
==================================================================================================================================================================
| Id |            Operation            | Name  |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read | Read  | Activity |       Activity Detail       |
|    |                                 |       | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs | Bytes |   (%)    |         (# samples)         |
==================================================================================================================================================================
|  0 | SELECT STATEMENT                |       |         |       |         8 |     +2 |     1 |        1 |      |       |          |                             |
|  1 |   SORT AGGREGATE                |       |       1 |       |         8 |     +2 |     1 |        1 |      |       |          |                             |
|  2 |    NESTED LOOPS                 |       |         |       |         8 |     +2 |     1 |     4214 |      |       |          |                             |
|  3 |     NESTED LOOPS                |       |    5025 | 11890 |         8 |     +2 |     1 |     4214 |      |       |          |                             |
|  4 |      TABLE ACCESS FULL          | FACT  |    2408 |  3535 |         9 |     +1 |     1 |     2404 |  109 | 100MB |    11.11 | direct path read (1)        |
|  5 |      INDEX RANGE SCAN           | DIM_I |       2 |     1 |         8 |     +2 |  2404 |     4214 |   12 | 98304 |          |                             |
|  6 |     TABLE ACCESS BY INDEX ROWID | DIM   |       2 |     4 |         8 |     +2 |  4214 |     4214 | 3360 |  26MB |    88.89 | db file sequential read (8) |
==================================================================================================================================================================

четверг, 29 октября 2015 г.

12c: PX Auto DOP without IO Calibration

I have upgraded one of data warehouse database this week and noticed changes in behaviour of PX Auto DOP feature.
According to the documentation of the parallel_degree_policy parameter Oracle Database Reference 11g:
  • MANUAL
    Disables automatic degree of parallelism, statement queuing, and in-memory parallel execution. This reverts the behavior of parallel execution to what it was prior to Oracle Database 11g Release 2 (11.2). This is the default.
  • LIMITED
    Enables automatic degree of parallelism for some statements but statement queuing and in-memory Parallel Execution are disabled. Automatic degree of parallelism is only applied to those statements that access tables or indexes decorated explicitly with the DEFAULT degree of parallelism using the PARALLEL clause. Statements that do not access any tables or indexes decorated with the DEFAULT degree of parallelism will retain the MANUAL behavior.
  • AUTO
    Enables automatic degree of parallelism, statement queuing, and in-memory parallel execution.
In short:
MANUAL - disables Auto DOP, parallel statement queuing, and in-memory parallel execution.
AUTO - enables all of the above.
What if you need parallel statement queuing but dont need Auto DOP?
MOS note suggests to add hint STATEMENT_QUEUING in your statements manually: How to Achieve Parallel Statement Queuing for an SQL When PARALLEL_DEGREE_POLICY=MANUAL (Doc ID 1902069.1)
In 11g there was an clever automatic way to achieve this:
VLDB and Partitioning Guide 11g
When PARALLEL_DEGREE_POLICY is set to AUTO, Oracle Database determines whether the statement should run in parallel based on the cost of the operations in the execution plan and the hardware characteristics.
The hardware characteristics include I/O calibration statistics so these statistics must be gathered otherwise Oracle Database does not use the automatic degree policy feature.

So I simply installed PARALLEL_DEGREE_POLICY=AUTO and didn't gather IO calibration statistics.
This prevents Auto DOP but parallel statement queuing and in-memory parallel execution still enabled.
Now in 12c things got changed:
VLDB and Partitioning Guide 12c
When PARALLEL_DEGREE_POLICY is set to AUTO, Oracle Database determines whether the statement should run in parallel based on the cost of the operations in the execution plan and the hardware characteristics.
The hardware characteristics include I/O calibration statistics so these statistics should be gathered.
If I/O calibration is not run to gather the required statistics, a default calibration value is used to calculate the cost of operations and the degree of parallelism.

Have you seen the difference?
It means that now Auto DOP works without IO calibration statistics.
I dont see any announcement in the New Features Guide or VLDB and Partitioning Guide about this change.
Yes, I agree that configuration with PARALLEL_DEGREE_POLICY=AUTO without IO calibration statistics looks an unusual thing.
But I would prefer that Oracle mentions such things at least in the "Changes in this release" section in the relevant book.
I.e. here.

среда, 30 сентября 2015 г.

Bug 20214168 - Wrong Results using aggregations of CASE expression with fix of bug 20003240 present

I started to use Oracle 12c in the early 2014 in non-production environments.
At now I am extensively testing the ADO, In-Memory functionalities and have couple of database in production.
They already use DBBP (Database Bundle Patch) 10 for engineering systems and database In-Memory and have several underscope parameters in effect.
Unfortunately, a few days ago we have faced with the wrong results caused by the Bug 20214168 - Wrong Results using aggregations of CASE expression with fix of bug 20003240 present.
This problem was already discussed on the freelist oracle-l thread Strange Behaviour (with Test Case).
Original thread was Exadata specific.
We dont have an Exadata yet but we use DBIM (Database In-Memory) and so we install DBBP patches.
I found out that solution from the thread with event 10055 don't help to resolve wrong results issue in our environment.
Let's create test table and data from the original bug:
SQL> create table test_fact(chrtype varchar2(3), rate number);
SQL> insert into test_fact values('R03', 1.3);
SQL> insert into test_fact values('LDU', 0.21);
SQL> 
SQL> select *
  2    from test_fact;

CHRTYPE         RATE
--------- ----------
R03              1.3
LDU              .21
Now execute below query:
SQL> select sum(
  2           case
  3             when chrtype in ('R03', 'LDU')
  4             then rate/10
  5           end) result
  6    from test_fact;

    RESULT
----------
      .041
Notice wrong results. Above query should return 0.151 and not 0.041.
Now we will execute same query with the event 10055 set:
SQL> alter session set events '10055 trace name context forever, level 0x200';
SQL> 
SQL> select sum(
  2           case
  3             when chrtype in ('R03', 'LDU')
  4             then rate/10
  5           end) result
  6    from test_fact;

    RESULT
----------
      .041
Notice that results are the same (they are wrong) and the event 10055 set to level 0x200 dont help.
I observed these results in the environment with Patch 21188742 - Database Patch for Engineered Systems and DB In-Memory 12.1.0.2.10 (Jul2015) applied.
May be other levels of event 10055 could help?
[oracle@localhost ~]$ oerr ora 10055
10055, 00000, "Rowsets: turn off rowsets for various operations"
// *Document: NO
// *Cause:    N/A
// *Action:   Turns off rowsets for various operations
//            Level:
//            0x00000001 - turn off for table scan
//            0x00000002 - turn off for hash join consume
//            0x00000004 - turn off for hash join produce
//            0x00000008 - turn off for group by
//            0x00000010 - turn off for sort
//            0x00000020 - turn off for table-queue out
//            0x00000040 - turn off for table-queue in
//            0x00000080 - turn off for identity
//            0x00000100 - turn off for granule iterator
//            0x00000200 - turn off for EVA functions
//            0x00000400 - turn off for PL/SQL
//            0x00000800 - turn off for upgrade
//            0x00001000 - turn off for database startup
//            0x00002000 - turn off for blobs and clobs
//            0x00004000 - turn off for tracing row source
//            0x00008000 - turn off rowset information in explain plan
//            0x00010000 - disable hash join rowsets fast path
//            0x00020000 - turn off for bloom create
//            0x00040000 - turn off for bloom use
//            0x00080000 - disable prefetch for hash join
//            0x00100000 - disable prefetch for bloom
//            0x00200000 - disable semi blocking hash join
//            0x00400000 - turn off rowset for fixed table
//
Unfortunately, other levels also have not helped.
I found 2 possible solutions for this problem case.
1. set _rowsets_enabled to false:
SQL> select sum(
  2           case
  3             when chrtype in ('R03', 'LDU')
  4             then rate/10
  5           end) result
  6    from test_fact;

    RESULT
----------
      .041
SQL> 
SQL> alter session set "_rowsets_enabled"=false;
SQL> 
SQL> select sum(
  2           case
  3             when chrtype in ('R03', 'LDU')
  4             then rate/10
  5           end) result
  6    from test_fact;

    RESULT
----------
      .151
2. set _rowsets_max_rows to 1:
SQL> select sum(
  2           case
  3             when chrtype in ('R03', 'LDU')
  4             then rate/10
  5           end) result
  6    from test_fact;

    RESULT
----------
      .041
SQL> 
SQL> alter session set "_rowsets_max_rows"=1;
SQL> 
SQL> select sum(
  2           case
  3             when chrtype in ('R03', 'LDU')
  4             then rate/10
  5           end) result
  6    from test_fact;

    RESULT
----------
      .151
Off course, we shouldn't use underscope parameters without Oracle Support agreement.
Instead we should apply the fix for bug 20214168 ASAP.
I tried to apply the patch 20214168 to this Oracle Home but OPatch informed me that patch already applied!
Recommended actions: The fixes by this patch are currently in the Oracle Home. There is no need to apply this patch.
Patch : 20214168

        Bug SubSet of 21125181
        Subset bugs are:
        20214168
From the opatch lsinv I saw that definitely fix for bug 20214168 already installed:
Local Machine Information::
Hostname: localhost
ARU platform id: 23
ARU platform description:: Solaris Operating System (SPARC 64-bit)
                
Installed Top-level Products (1): 
Oracle Database 12c                                                  12.1.0.2.0There are 1 products installed in this Oracle Home.
Interim patches (5) :
Patch  20831113     : applied on Wed Aug 05 15:01:24 NOVT 2015
Unique Patch ID:  18927529
Patch description:  "OCW Patch Set Update : 12.1.0.2.4 (20831113)"
   Created on 23 Jun 2015, 06:58:08 hrs UTC
   Bugs fixed:
     18589889, 19139608, 19280860, 19061429, 19133945, 19341538, 20011424
...skip...
Patch  21125181     : applied on Wed Aug 05 14:50:35 NOVT 2015
Unique Patch ID:  19005983
Patch description:  "DATABASE BUNDLE PATCH: 12.1.0.2.10 (21125181)"
   Created on 1 Jul 2015, 22:01:24 hrs PST8PDT
Sub-patch  20594149; "DATABASE BUNDLE PATCH: 12.1.0.2.7 (20594149)"
Sub-patch  20415006; "DATABASE BUNDLE PATCH: 12.1.0.2.6 (20415006)"
Sub-patch  20243804; "DATABASE BUNDLE PATCH: 12.1.0.2.5 (20243804)"
   Bugs fixed:
...skip...
     19990543, 19012044, 20214168, 20209481, 18885870, 13640676, 13498243
...skip...
Patch  19396455     : applied on Mon Jul 13 17:43:01 NOVT 2015
Unique Patch ID:  18154832
   Created on 15 Oct 2014, 20:19:28 hrs PST8PDT
   Bugs fixed:
     19396455
Patch  19567916     : applied on Mon Jul 13 17:41:37 NOVT 2015
Unique Patch ID:  18878751
   Created on 1 May 2015, 01:31:51 hrs PST8PDT
   Bugs fixed:
     19567916
Patch  20879889     : applied on Mon Jul 13 17:33:26 NOVT 2015
Unique Patch ID:  18969474
   Created on 27 May 2015, 10:30:29 hrs PST8PDT
   Bugs fixed:
     20879889
I have found that bug 21214168 incorrectly informed as fixed but actually it's not fixed: Bug 21553476 - Wrong Results using aggregations of CASE expression with fix of bug 20003240 present in Exadata (Doc ID 21553476.8).
So instead of install patch 21214168, we should install Patch 21553476: EXADATA X5-2 RESULTS WRONG NUMERIC CALCULATION.
I had installed patch 21553476 and problem was resolved after that!
SQL> select sum(
  2           case
  3             when chrtype in ('R03', 'LDU')
  4             then rate/10
  5           end) result
  6    from test_fact
  7  /

    RESULT
----------
      .151

понедельник, 17 августа 2015 г.

EM 12.1.0.4 Installing recommended Plug-Ins and Patches

In one of the project we will use DBaaS (Database as a Service) functionality of EM Cloud Control 12c.
We faced with a multiple issue and decided to upgrade EM and install Oracle recommended patches from note:
Enterprise Manager Cloud Control 12c Recommended Plug-Ins and Patches for Database as a Service (DBaaS) (Doc ID 1549855.1)
Specifically:
Patch 20870437: ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU (GENERIC)
and
Patch 21415432: enterprise manager for oms plugins 12.1.0.4.14
We had vanilla EM 12.1.0.4 installation without any PSU or patches (except Cloud Framework Plug-In version 12.1.0.2.0).

ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU
At the step of installation:
2.9 Run the following command on each OMS instance home to ensure that you do not have any issues with configuration and binary prerequisite checks,
I encountered an error:
[oracle@mgmt 20870437]$ 
/u01/app/oracle/oms12/oms/OPatch/opatchauto apply -analyze 
-property_file /u01/app/oracle/oms12/oms/OPatch/wlskeys/property_file 
-invPtrLoc /u01/app/oracle/oms12/oms/oraInst.loc  
OPatchAuto.OMS_DISABLE_HOST_CHECK=true
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.
  
  
OPatchauto version : 11.1.0.10.4
OUI version        : 11.1.0.12.0
Running from       : /u01/app/oracle/oms12/oms
Log file location  : /u01/app/oracle/oms12/oms/cfgtoollogs/opatch/opatch2015-08-07_17-00-41PM_1.log
  
OPatchauto log file: /u01/app/oracle/oms12/oms/cfgtoollogs/opatchauto/20870437/opatch_oms_2015-08-07_17-00-43PM_analyze.log
  
  
  
OPatchauto
 failed to establish JMX connection to weblogic server. This could be 
because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3.
 Virtual host configuration. If OMS, weblogic server are on virtual host
 configuration, Please make sure to add 
OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. 
(example: /u01/app/oracle/oms12/oms/OPatch/opatchauto apply -analyze 
-property_file /u01/app/oracle/oms12/oms/OPatch/wlskeys/property_file 
-invPtrLoc /u01/app/oracle/oms12/oms/oraInst.loc 
OPatchAuto.OMS_DISABLE_HOST_CHECK=true 
OPatchAuto.OMS_DISABLE_HOST_CHECK=true)
  
Please check above conditions and if error(s) still persist, Please contact Oracle support.
  
  
[ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs.
OPatchauto failed:
OPatchauto
 failed to establish JMX connection to weblogic server. This could be 
because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3.
 Virtual host configuration. If OMS, weblogic server are on virtual host
 configuration, Please make sure to add 
OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. 
(example: /u01/app/oracle/oms12/oms/OPatch/opatchauto apply -analyze 
-property_file /u01/app/oracle/oms12/oms/OPatch/wlskeys/property_file 
-invPtrLoc /u01/app/oracle/oms12/oms/oraInst.loc 
OPatchAuto.OMS_DISABLE_HOST_CHECK=true 
OPatchAuto.OMS_DISABLE_HOST_CHECK=true)
  
Please check above conditions and if error(s) still persist, Please contact Oracle support.
  
Log file location: /u01/app/oracle/oms12/oms/cfgtoollogs/opatchauto/20870437/opatch_oms_2015-08-07_17-00-43PM_analyze.log
  
Recommended actions: Please correct the interview inputs and run opatchauto again.
  
OPatchauto failed with error code 231
[oracle@mgmt 20870437]$
I searched through MOS knowledge base and have decided that this error due to the old OPatch version.
I found a note with a same error:
EM12c : OPatchauto fails with : opatchauto finds that weblogic credentials may not be right or admin server is down (Doc ID 1644317.1)
From which I concluded that issue should be resolved after OPatch 11.1.0.11 install.
As I seen from the logs, we have used OPatch version 11.1.0.10.4.
I installed latest OPatch version 11.1.0.x and installation of PSU was ended without any issues.

Enterprise Manager for OMS Plug-Ins 12.1.0.4.14
opatchauto apply -analyze command showed that from 18 sub-patches I could install only 1.
[oracle@mgmt 21415432]$ /u01/app/oracle/oms12/oms/OPatch/opatchauto apply -analyze -invPtrLoc /u01/app/oracle/oms12/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.
 
OPatchauto version : 11.1.0.12.3
OUI version        : 11.1.0.12.0
Running from       : /u01/app/oracle/oms12/oms
Log file location  : /u01/app/oracle/oms12/oms/cfgtoollogs/opatch/opatch2015-08-10_15-52-13PM_1.log
 
OPatchauto log file: /u01/app/oracle/oms12/oms/cfgtoollogs/opatchauto/21415432/opatch_oms_2015-08-10_15-52-16PM_analyze.log
 
Please enter OMS weblogic admin server URL(t3s://mgmt.localdomain:7102):>
Please enter OMS weblogic admin server username:> weblogic
Please enter OMS weblogic admin server password:>
 
WARNING: Sub-patch "19060193" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "19513319" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "19512418" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "20466650" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "20623942" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "20950018" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "20950277" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21167953" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21167965" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21167991" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21174690" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21185095" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324604" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324646" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324654" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324861" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324890" skipped for incompatibility with components installed in the OMS system.
 
Configuration Validation: Success
 
Running apply prerequisite checks for sub-patch(es) "21167573" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.cfw.oms.plugin_12.1.0.2.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.cfw.oms.plugin_12.1.0.2.0/cfgtoollogs/opatch/21167573_Aug_10_2015_15_52_13/ApplyPrereq2015-08-10_15-53-08PM_2.log
Sub-patch(es) "21167573" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.cfw.oms.plugin_12.1.0.2.0"
 
Complete Summary
================
 
All log file names referenced below can be accessed from the directory "/u01/app/oracle/oms12/oms/cfgtoollogs/opatch/2015-08-10_15-52-13PM_SystemPatch_21415432_1"
 
Prerequisites analysis summary:
-------------------------------
  
The following sub-patch(es) are applicable:
 
  Oracle Home Name   Sub-patches                                          Log file
  ----------------   -----------                                          --------
         OraHome18      21167573   21167573_ApplyPrereq2015-08-10_15-53-08PM_2.log
 
The following sub-patches are incompatible with components installed in the OMS system:
19060193,19513319,19512418,20466650,20623942,20950018,20950277,21167953,21167965,21167991,21174690,21185095,21324604,21324646,21324654,21324861,21324890
 
--------------------------------------------------------------------------------
 
The following warnings have occurred during OPatchauto execution:
1)  Sub-patch "19060193" skipped for incompatibility with components installed in the OMS system.
2)  Sub-patch "19513319" skipped for incompatibility with components installed in the OMS system.
3)  Sub-patch "19512418" skipped for incompatibility with components installed in the OMS system.
4)  Sub-patch "20466650" skipped for incompatibility with components installed in the OMS system.
5)  Sub-patch "20623942" skipped for incompatibility with components installed in the OMS system.
6)  Sub-patch "20950018" skipped for incompatibility with components installed in the OMS system.
7)  Sub-patch "20950277" skipped for incompatibility with components installed in the OMS system.
8)  Sub-patch "21167953" skipped for incompatibility with components installed in the OMS system.
9)  Sub-patch "21167965" skipped for incompatibility with components installed in the OMS system.
10)  Sub-patch "21167991" skipped for incompatibility with components installed in the OMS system.
11)  Sub-patch "21174690" skipped for incompatibility with components installed in the OMS system.
12)  Sub-patch "21185095" skipped for incompatibility with components installed in the OMS system.
13)  Sub-patch "21324604" skipped for incompatibility with components installed in the OMS system.
14)  Sub-patch "21324646" skipped for incompatibility with components installed in the OMS system.
15)  Sub-patch "21324654" skipped for incompatibility with components installed in the OMS system.
16)  Sub-patch "21324861" skipped for incompatibility with components installed in the OMS system.
17)  Sub-patch "21324890" skipped for incompatibility with components installed in the OMS system.
--------------------------------------------------------------------------------
 
OPatchauto Session completed with warnings.
 
Log file location: /u01/app/oracle/oms12/oms/cfgtoollogs/opatchauto/21415432/opatch_oms_2015-08-10_15-52-16PM_analyze.log
 
OPatchauto completed with warnings.
 
[oracle@mgmt 21415432]$
Most of the patches was skipped due to incompatibility checks:
WARNING: Sub-patch "21324890" skipped for incompatibility with components installed in the OMS system.
I searched through MOS knowledge base and found a note:
EM 12c: Applying a System Patch to the Enterprise Manager 12.1.0.4 Cloud Control OMS Oracle Home Fails at Analyze with Error: Sub-patch skipped for incompatibility with components installed in the OMS system (Doc ID 1921337.1) which tell us that:

Such incompatibility errors result from two scenarios:
Case 1:The Bundle Patch is comprised of patches for the OMS Oracle Home as well as Plug-in Homes. If one of these Plugins is not present in the environment, then these patches would be skipped automatically as they are not needed.
Case 2: If the plug-in is present but is of a lower version than the plug-in versions mentioned above, then such errors can also result.


Patch 21415432 (Enterprise Manager for OMS Plug-Ins 12.1.0.4.14) contains patches for the below list of plugins:
  • Enterprise Manager for Cloud: 12.1.0.8.0
  • Enterprise Manager for Fusion Middleware: 12.1.0.6.0
  • Enterprise Manager for Oracle Database: 12.1.0.6.0
  • Enterprise Manager for Fusion Applications: 12.1.0.6.0
  • Enterprise Manager for Storage Management Framework: 12.1.0.4.0
  • Enterprise Manager for Siebel: 12.1.0.5.0
  • Enterprise Manager for Exadata : 12.1.0.6.0
  • Enterprise Manager for Oracle Virtual Infrastructure: 12.1.0.1.0
  • Enterprise Manager for Virtualization: 12.1.0.6.0
  • Enterprise Manager for My Oracle Support: 12.1.0.6.0
  • Enterprise Manager for Chargeback and Capacity Planning: 12.1.0.6.0
  • Enterprise Manager for Apache Tomcat: 12.1.0.3.0
  • Enterprise Manager for Oracle Database: 12.1.0.7.0
  • Enterprise Manager for Storage Management Framework: 12.1.0.5.0
  • Enterprise Manager for Cloud: 12.1.0.9.0
  • Enterprise Manager for Fusion Middleware: 12.1.0.7.0
Reading further the note
EM 12c: Applying a System Patch to the Enterprise Manager 12.1.0.4 Cloud Control OMS Oracle Home Fails at Analyze with Error: Sub-patch skipped for incompatibility with components installed in the OMS system (Doc ID 1921337.1), I have found the emcli commands which can be used to verify versions of plugins installed on the OMS:
./emcli login -username=sysman
./emcli sync
./emcli list_plugins_on_server
I run above commands and found that I have plugin versions which should be patched by the Patch 21415432 (I highlight DB and SMF plugins for simplicity):
OMS name is mgmt.localdomain:4889_Management_Service
Plug-in Name                        Plugin-id                   Version [revision]
Oracle Cloud Framework              oracle.sysman.cfw           12.1.0.2.0
Oracle Database                     oracle.sysman.db            12.1.0.6.0
Oracle Fusion Middleware            oracle.sysman.emas          12.1.0.6.0
Oracle MOS (My Oracle Support)      oracle.sysman.mos           12.1.0.6.0
Oracle Storage Management Framework oracle.sysman.smf           12.1.0.4.0
Oracle Cloud Application            oracle.sysman.ssa           12.1.0.8.0
Oracle Virtual Infrastructure       oracle.sysman.vi            12.1.0.1.0
Oracle Virtualization               oracle.sysman.vt            12.1.0.6.0
Oracle Exadata                      oracle.sysman.xa            12.1.0.6.0
I continued searching through MOS and found the Bug 21287619 : OPATCHAUTO UNABLE TO IDENTIFY INSTALLED PLUGIN_HOME which was without the resolution but provided some ideas for further investigations.
I started to check them.
At first, I verified that I have correct plugin versions in PLUGIN_HOME:
[oracle@mgmt 21415432]$ ls -la /u01/app/oracle/oms12/plugins/
drwxr-xr-x 14 oracle oinstall 4096 Aug  4 15:05 .
drwxr-x--- 16 oracle oinstall 4096 Jul 11  2014 ..
drwxr-xr-x  2 oracle oinstall 4096 Aug  4 15:31 backup
drwxr-xr-x 12 oracle oinstall 4096 Jul 11  2014 oracle.sysman.cfw.oms.plugin_12.1.0.1.0
drwxr----- 14 oracle oinstall 4096 Aug 10 14:58 oracle.sysman.cfw.oms.plugin_12.1.0.2.0
drwxr-xr-x 18 oracle oinstall 4096 Aug  4 15:03 oracle.sysman.db.oms.plugin_12.1.0.6.0
drwxr-xr-x 18 oracle oinstall 4096 Jul 11  2014 oracle.sysman.emas.oms.plugin_12.1.0.6.0
drwxr-xr-x 13 oracle oinstall 4096 Jul 11  2014 oracle.sysman.mos.oms.plugin_12.1.0.6.0
drwxr-xr-x 12 oracle oinstall 4096 Jul 11  2014 oracle.sysman.smf.oms.plugin_12.1.0.4.0
drwxr-xr-x 14 oracle oinstall 4096 Jul 11  2014 oracle.sysman.ssa.oms.plugin_12.1.0.8.0
drwxr-xr-x 12 oracle oinstall 4096 Jul 11  2014 oracle.sysman.vi.oms.plugin_12.1.0.1.0
drwxr-xr-x 15 oracle oinstall 4096 Jul 11  2014 oracle.sysman.vt.oms.plugin_12.1.0.6.0
drwxr-xr-x 13 oracle oinstall 4096 Jul 11  2014 oracle.sysman.xa.oms.plugin_12.1.0.6.0
drwxr-xr-x  2 oracle oinstall 4096 Jul 11  2014 tmp
I reviewed opatch/opatchauto lspatches output:
[oracle@mgmt 21415432]$ opatch lspatches
20870437;ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU (GENERIC)
[oracle@mgmt 21415432]$ opatchauto lspatches
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.
Oracle Home:/u01/app/oracle/oms12/oms
20870437;ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU (GENERIC)
The following groups of patch(es) are applied as System Patch bundle(s):
20870437
For more details on installed patch(es) in platform, plugin homes, Please do "/u01/app/oracle/oms12/oms/OPatch/opatch lsinventory -details -oh ".
opatch lsinv output:
[oracle@mgmt 21415432]$ opatch lsinv
Oracle Interim Patch Installer version 11.1.0.12.7
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/oracle/oms12/oms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/oms12/oms/oraInst.loc
OPatch version    : 11.1.0.12.7
OUI version       : 11.1.0.12.0
Log file location : /u01/app/oracle/oms12/oms/cfgtoollogs/opatch/opatch2015-08-11_08-41-48AM_1.log
OPatch detects the Middleware Home as "/u01/app/oracle/oms12"
Lsinventory Output file location : /u01/app/oracle/oms12/oms/cfgtoollogs/opatch/lsinv/lsinventory2015-08-11_08-41-48AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
EM Platform (OMS)                                                    12.1.0.4.0
There are 1 products installed in this Oracle Home.
Interim patches (1) :
Patch  20870437     : applied on Mon Aug 10 15:19:39 MSK 2015
Unique Patch ID:  18978384
Patch description:  "ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.4 PSU (GENERIC)"
   Created on 2 Jul 2015, 03:07:25 hrs PST8PDT
   Bugs fixed:
     20010360, 20346629, 18599481, 19459173, 20646440, 18830050, 18244680
     19614772, 20492788, 18932882, 19355307, 20263906, 20556489, 20106145
     14301574, 18721562, 19328391, 19535114, 19861538, 19794113, 19604817
     20596437, 19892848, 19218456, 19978077, 18505219, 17843191, 20345612
     20748748, 20024779, 18256983, 19329809, 19300103, 17987955, 18351725
     18903783, 19430853, 19845101, 20870437, 20223681, 20019184, 20263188
     19069940, 19861412, 19136663, 18446798, 19474685, 18765545, 18038059
     17783148, 18041873, 19509780, 19030447, 20933667, 19574756, 19258481
     18938561, 17812648, 20377474, 19572261, 18945007, 20307428, 18388324
     19500609, 17991694, 19137411, 19474929, 19592446, 18710124, 20536095
     19352118, 19557054, 18726708, 19595813, 19170861, 19685443, 18353162
     19289270, 19367214, 18906451, 19458672, 20284029, 18490444, 19766881
     18509136, 20739088, 19511544, 18695961, 19812966, 20576148, 20623258
     19055251, 20260177, 19323634, 19941819, 18725891, 19824587, 19780847
     18359621, 19393211, 19211048, 20505155, 18476364, 18490468, 18247849
     19071602, 19068674, 18490455, 19524568, 19061168, 19077861, 20031287
     18756725, 19281252, 18977955, 18521731, 18485889, 19775544, 18900197
     17834366, 20460507, 19545505, 19823996, 20645335, 20633770, 17407918
     18706824, 20568225, 18805935, 19376082, 19146514, 18490432, 19694744
     18230284, 18726208, 20044313, 18516298, 20642633, 18999400, 20692416
     20392036, 19362386, 19452280, 18356966, 20617058, 19055979, 19830994
     20030493, 19823761, 18834216, 19196979, 19785223, 18389542
  
--------------------------------------------------------------------------------
OPatch succeeded.
I found an answer in an inventory.xml from Central Inventory:
<HOME NAME="oms12c1" LOC="/u01/app/oracle/oms12/oms" TYPE="O" IDX="3">
   <REFHOMELIST>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.4.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.xa.oms.plugin_12.1.0.4.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.emas.oms.plugin_12.1.0.4.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.mos.oms.plugin_12.1.0.5.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/jdk16"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.vt.oms.plugin_12.1.0.5.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.ssa.oms.plugin_12.1.0.6.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.2.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.ssa.oms.plugin_12.1.0.7.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.5.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.3.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sun.oss7.oms.plugin_12.1.0.4.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.emas.oms.plugin_12.1.0.5.0"/>
      <REFHOME LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.cfw.oms.plugin_12.1.0.2.0"/>
   </REFHOMELIST>
</HOME>
Notice, that inventory.xml contains old plugin versions! It's hard to say what exactly leads to a such behaviour and I didn't have enough time for further investigations.
It was enough for me, that inventory.xml dont contain actual plugin versions.

Example 1: oracle.sysman.smf.oms.plugin (Storage Management Framework).
emcli list_plugins_on_server shows version 12.1.0.4:
Oracle Storage Management Framework          oracle.sysman.smf             12.1.0.4.0
inventory.xml contains information about versions 12.1.0.2/12.1.0.3:
<HOME NAME="OraHome13" LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.2.0" TYPE="O" IDX="24" PLUGIN="T">
<HOME NAME="OraHome16" LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.3.0" TYPE="O" IDX="29" PLUGIN="T">
Patch 21415432 contains fixes for SMF plugin versions 12.1.0.4/12.1.0.5.

Example 2: oracle.sysman.db.oms.plugin (Database Plugin).
emcli list_plugins_on_server shows version 12.1.0.6:
Oracle Database                               oracle.sysman.db                12.1.0.6.0
inventory.xml contains information about versions 12.1.0.4/12.1.0.5:
<HOME NAME="oracle_sysman_db11" LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.4.0" TYPE="O" IDX="4" PLUGIN="T">
<HOME NAME="OraHome15" LOC="/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.5.0" TYPE="O" IDX="28" PLUGIN="T">
Patch 21415432 contains fixes for Database Plugin versions 12.1.0.6/12.1.0.7.
Due to such inconsistencies opatch reported a message like below:
WARNING: Sub-patch "19060193" skipped for incompatibility with components installed in the OMS system.
I decided to recover lost inventory entities by note:
EM 12c: How to Recover a Lost Inventory Record for the Enterprise Manager 12.1.0.1 Cloud Control OMS and Management Agent (Doc ID 1480237.1)

GOAL

In the case that recovering a lost inventory record may be necessary, this note describes how to accomplish this for the Enterprise Manager 12.1.0.1 Cloud Control OMS and Management Agent.


First of all, I had take backup of Central Inventory, local inventory, ORACLE_HOME of OMS.
8 plugins from 9 total didn't have an actual version in the inventory.xml.
For each of them, I executed steps from the note:
EM 12c: How to Recover a Lost Inventory Record for the Enterprise Manager 12.1.0.1 Cloud Control OMS and Management Agent (Doc ID 1480237.1)
I remind, that before my actions only 1 sub-patch (from 18) could be applied to the OMS.
And after that, opatchauto analyze showed that 9 sub-patches from 18 could be applied to the OMS. Bingo!
OMS Plug-Ins patches contains fixes for 2 versions of each plugin. And only one of them was installed in my environment.
[oracle@mgmt 21415432]$ /u01/app/oracle/oms12/oms/OPatch/opatchauto apply -analyze -invPtrLoc /u01/app/oracle/oms12/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.
 
OPatchauto version : 11.1.0.12.3
OUI version        : 11.1.0.12.0
Running from       : /u01/app/oracle/oms12/oms
Log file location  : /u01/app/oracle/oms12/oms/cfgtoollogs/opatch/opatch2015-08-11_17-13-05PM_1.log
OPatchauto log file: /u01/app/oracle/oms12/oms/cfgtoollogs/opatchauto/21415432/opatch_oms_2015-08-11_17-13-08PM_analyze.log
 
Please enter OMS weblogic admin server URL(t3s://mgmt.localdomain:7102):>
Please enter OMS weblogic admin server username:> weblogic
Please enter OMS weblogic admin server password:>
 
WARNING: Sub-patch "19512418" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "20466650" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "20950277" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21167965" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21185095" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324604" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324646" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324654" skipped for incompatibility with components installed in the OMS system.
WARNING: Sub-patch "21324861" skipped for incompatibility with components installed in the OMS system.
  
Configuration Validation: Success
 
Running apply prerequisite checks for sub-patch(es) "21324890" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.vi.oms.plugin_12.1.0.1.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.vi.oms.plugin_12.1.0.1.0/cfgtoollogs/opatch/21324890_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-13-58PM_2.log
Sub-patch(es) "21324890" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.vi.oms.plugin_12.1.0.1.0"
 
Running apply prerequisite checks for sub-patch(es) "21167573" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.cfw.oms.plugin_12.1.0.2.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.cfw.oms.plugin_12.1.0.2.0/cfgtoollogs/opatch/21167573_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-14-01PM_2.log
Sub-patch(es) "21167573" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.cfw.oms.plugin_12.1.0.2.0"
 
Running apply prerequisite checks for sub-patch(es) "21174690" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.6.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.6.0/cfgtoollogs/opatch/21174690_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-14-03PM_2.log
Sub-patch(es) "21174690" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.db.oms.plugin_12.1.0.6.0"
 
Running apply prerequisite checks for sub-patch(es) "21167953" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.xa.oms.plugin_12.1.0.6.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.xa.oms.plugin_12.1.0.6.0/cfgtoollogs/opatch/21167953_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-14-06PM_2.log
Sub-patch(es) "21167953" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.xa.oms.plugin_12.1.0.6.0"
 
Running apply prerequisite checks for sub-patch(es) "19060193" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.vt.oms.plugin_12.1.0.6.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.vt.oms.plugin_12.1.0.6.0/cfgtoollogs/opatch/19060193_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-14-09PM_2.log
Sub-patch(es) "19060193" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.vt.oms.plugin_12.1.0.6.0"
 
Running apply prerequisite checks for sub-patch(es) "20950018" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.emas.oms.plugin_12.1.0.6.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.emas.oms.plugin_12.1.0.6.0/cfgtoollogs/opatch/20950018_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-14-11PM_2.log
Sub-patch(es) "20950018" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.emas.oms.plugin_12.1.0.6.0"
 
Running apply prerequisite checks for sub-patch(es) "20623942" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.ssa.oms.plugin_12.1.0.8.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.ssa.oms.plugin_12.1.0.8.0/cfgtoollogs/opatch/20623942_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-14-14PM_2.log
Sub-patch(es) "20623942" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.ssa.oms.plugin_12.1.0.8.0"
 
Running apply prerequisite checks for sub-patch(es) "19513319" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.4.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.4.0/cfgtoollogs/opatch/19513319_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-14-17PM_2.log
Sub-patch(es) "19513319" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.smf.oms.plugin_12.1.0.4.0"
 
Running apply prerequisite checks for sub-patch(es) "21167991" and Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.mos.oms.plugin_12.1.0.6.0"...
Please monitor OPatch log file: /u01/app/oracle/oms12/plugins/oracle.sysman.mos.oms.plugin_12.1.0.6.0/cfgtoollogs/opatch/21167991_Aug_11_2015_17_13_05/ApplyPrereq2015-08-11_17-14-19PM_2.log
Sub-patch(es) "21167991" are successfully analyzed for Oracle Home "/u01/app/oracle/oms12/plugins/oracle.sysman.mos.oms.plugin_12.1.0.6.0"
 
Complete Summary
================
 
All log file names referenced below can be accessed from the directory "/u01/app/oracle/oms12/oms/cfgtoollogs/opatch/2015-08-11_17-13-05PM_SystemPatch_21415432_1"
 
Prerequisites analysis summary:
-------------------------------
 
The following sub-patch(es) are applicable:
 
                                 Oracle Home Name   Sub-patches                                          Log file
                                 ----------------   -----------                                          --------
                       oracle_home_MOS_12_1_0_6_0      21167991   21167991_ApplyPrereq2015-08-11_17-14-19PM_2.log
  oracle_home_Oracle_Fusion_Middleware_12_1_0_6_0      20950018   20950018_ApplyPrereq2015-08-11_17-14-11PM_2.log
                        oracle_home_OV_12_1_0_6_0      19060193   19060193_ApplyPrereq2015-08-11_17-14-09PM_2.log
                                        OraHome18      21167573   21167573_ApplyPrereq2015-08-11_17-14-01PM_2.log
                       oracle_home_OVI_12_1_0_1_0      21324890   21324890_ApplyPrereq2015-08-11_17-13-58PM_2.log
                        oracle_home_OE_12_1_0_6_0      21167953   21167953_ApplyPrereq2015-08-11_17-14-06PM_2.log
                       oracle_home_OCA_12_1_0_8_0      20623942   20623942_ApplyPrereq2015-08-11_17-14-14PM_2.log
           oracle_home_Oracle_Database_12_1_0_6_0      21174690   21174690_ApplyPrereq2015-08-11_17-14-03PM_2.log
                       oracle_home_SMF_12_1_0_4_0      19513319   19513319_ApplyPrereq2015-08-11_17-14-17PM_2.log
  
The following sub-patches are incompatible with components installed in the OMS system:
19512418,20466650,20950277,21167965,21185095,21324604,21324646,21324654,21324861
 
--------------------------------------------------------------------------------
 
The following warnings have occurred during OPatchauto execution:
1)  Sub-patch "19512418" skipped for incompatibility with components installed in the OMS system.
2)  Sub-patch "20466650" skipped for incompatibility with components installed in the OMS system.
3)  Sub-patch "20950277" skipped for incompatibility with components installed in the OMS system.
4)  Sub-patch "21167965" skipped for incompatibility with components installed in the OMS system.
5)  Sub-patch "21185095" skipped for incompatibility with components installed in the OMS system.
6)  Sub-patch "21324604" skipped for incompatibility with components installed in the OMS system.
7)  Sub-patch "21324646" skipped for incompatibility with components installed in the OMS system.
8)  Sub-patch "21324654" skipped for incompatibility with components installed in the OMS system.
9)  Sub-patch "21324861" skipped for incompatibility with components installed in the OMS system.
--------------------------------------------------------------------------------
OPatchauto Session completed with warnings.
 
Log file location: /u01/app/oracle/oms12/oms/cfgtoollogs/opatchauto/21415432/opatch_oms_2015-08-11_17-13-08PM_analyze.log
 
OPatchauto completed with warnings.
Further installation was ended without any issues.