Sunday 22 February 2015

Query contents of Trace files


There are a number of ways to make trace files available to developers, but one simple way is to allow them query the contents of a trace file from SQL*Plus and spool it to a file on their local PC.
The following example creates a locked user with the necessary code to read the contents of specified trace files. This code is then made accessible to public and a directory object pointing to the trace location is created.

CONN / AS SYSDBA

-- Create locked user to hold code.
CREATE USER trace_user IDENTIFIED BY trace_user ACCOUNT LOCK;

-- Create a type and pipelined table function to read the specified trace file.
CREATE TYPE trace_user.t_trace_tab AS TABLE OF VARCHAR2(32767);
/

CREATE OR REPLACE FUNCTION trace_user.get_trace_file (p_trace_file IN VARCHAR2)
  RETURN trace_user.t_trace_tab PIPELINED
AS
  l_file UTL_FILE.file_type; 
  l_text VARCHAR2(32767);
BEGIN
  l_file := UTL_FILE.fopen('TRACE_DIR', p_trace_file, 'r', 32767);
  BEGIN
    LOOP
      UTL_FILE.get_line(l_file, l_text);
      PIPE ROW (l_text);
    END LOOP;
  EXCEPTION
    WHEN NO_DATA_FOUND THEN
      NULL;
  END;
  UTL_FILE.fclose(l_file);
  RETURN;
EXCEPTION
  WHEN OTHERS THEN
    PIPE ROW ('ERROR: ' || SQLERRM);
    IF UTL_FILE.is_open(l_file) THEN
      UTL_FILE.fclose(l_file);
    END IF;
    RETURN;
END;
/

-- Make the code accessible to PUBLIC.
GRANT EXECUTE ON trace_user.t_trace_tab TO PUBLIC;
GRANT EXECUTE ON trace_user.get_trace_file TO PUBLIC;
CREATE PUBLIC SYNONYM get_trace_file FOR trace_user.get_trace_file;

-- Create a directory object to the trace location.
CREATE OR REPLACE DIRECTORY trace_dir AS '/u01/app/oracle/diag/rdbms/db11g/DB11G/trace/';
GRANT READ ON DIRECTORY trace_dir TO trace_user;
With that in place we can see how a developer would create and return a trace file to their local PC.
-- Make sure the user has access to DBMS_MONITOR.
CONN / AS SYSDBA
GRANT EXECUTE ON dbms_monitor TO hr;

CONN hr/hr

-- Identify the current trace file.
SELECT value
FROM   v$diag_info
WHERE  name = 'Default Trace File';

VALUE
--------------------------------------------------------------------------------
/u01/app/oracle/diag/rdbms/db11g/DB11G/trace/DB11G_ora_6309.trc

SQL>

-- Turn of tracing.
EXEC DBMS_MONITOR.session_trace_enable(waits=>TRUE, binds=>FALSE);

-- Do something you wish to trace.
SET LINESIZE 100
COLUMN employee_name FORMAT A30
COLUMN department_name FORMAT A20
COLUMN manager_name FORMAT A30

SELECT e.first_name || ' ' || e.last_name AS employee_name,
       d.department_name,
       e2.first_name || ' ' || e2.last_name AS manager_name
FROM   employees e
       JOIN departments d ON e.department_id = d.department_id
       JOIN employees e2 ON e.manager_id = e2.employee_id
WHERE  d.department_name = 'Finance'
ORDER BY d.department_name;

EMPLOYEE_NAME                  DEPARTMENT_NAME      MANAGER_NAME
------------------------------ -------------------- ------------------------------
Nancy Greenberg                Finance              Neena Kochhar
Luis Popp                      Finance              Nancy Greenberg
Jose Manuel Urman              Finance              Nancy Greenberg
Ismael Sciarra                 Finance              Nancy Greenberg
John Chen                      Finance              Nancy Greenberg
Daniel Faviet                  Finance              Nancy Greenberg

6 rows selected.

SQL> 

-- Turn off tracing.
EXEC DBMS_MONITOR.session_trace_disable;

-- Spool the contents of the relevant trace file to a local file.
SET PAGESIZE 0 FEEDBACK OFF TRIMSPOOL ON TAB OFF
SPOOL C:\tracefile.trc

SELECT *
FROM   TABLE(get_trace_file('DB11G_ora_6309.trc'));

SPOOL OFF
SET PAGESIZE 14 FEEDBACK ON

The developer can now use a local installation of TKPROF or SQL Developer to translate the trace file.


Reference: http://oracle-base.com/articles/misc/sql-trace-10046-trcsess-and-tkprof.php

SQL Plan Management in Oracle


How Does SQL Plan Management Work?

When a SQL statement is hard parsed, the cost based optimizer produces several execution plans and selects the one with the lowest cost. If a SQL plan baseline is present, the optimizer compares the plan it just produced with the plans in the SQL plan baseline. If a matching plan is found that is flagged as accepted the plan is used. If the SQL plan baseline doesn't contain an accepted plan matching the one it just created, the optimizer evaluates the accepted plans in the SQL plan baseline and uses the one with the lowest cost. If the execution plan originally produced by the optimizer has a lower cost than those present in the SQL plan baseline, it is added to the baseline as a not-accepted plan, so it is not used until it is verified not to cause a reduction in performance. If a system change affects all existing accepted plans, so they are considered non-reproducible, the optimizer will use the plan with the lowest cost.

Automatic Plan Capture:

The value of the OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES parameter, whose default value is FALSE, determines if the system should automatically capture SQL plan baselines. When set to TRUE, the system records a plan history for SQL statements. The first plan for a specific statement is automatically flagged as accepted. Alternative plans generated after this point are not used until it is verified they do not cause performance degradations. Plans with acceptable performance are added to the SQL plan baseline during the evolution phase.


  SQL> ALTER SYSTEM SET OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=TRUE;

  System altered.

Manual Plan Loading:



DECLARE
  l_plans_loaded  PLS_INTEGER;
BEGIN
  l_plans_loaded := DBMS_SPM.load_plans_from_sqlset(
    sqlset_name => 'my_sqlset');
END;
/


DECLARE
  l_plans_loaded  PLS_INTEGER;
BEGIN
  l_plans_loaded := DBMS_SPM.load_plans_from_cursor_cache(
    sql_id => '1abc93ed0123p');
END;
/

Evolving SQL Plan Baselines:

Evolving a SQL plan baseline is the process by which the optimizer determines if non-accepted plans in the baseline should be accepted. As mentioned previously, manually loaded plans are automatically marked as accepted, so manual loading forces the evolving process. When plans are loaded automatically, the baselines are evolved using the EVOLVE_SQL_PLAN_BASELINE function, which returns a CLOB reporting its results.

SET LONG 10000
SELECT DBMS_SPM.evolve_sql_plan_baseline(sql_handle => 'SYS_SQL_7b76323ad90440b9')
FROM  dual;



Steps for SQL access advisor usage

DECLARE
  taskname varchar2(30) := 'SQLACCESS3638195';
  task_desc varchar2(256) := 'SQL Access Advisor';
  task_or_template varchar2(30) := 'SQLACCESS_EMTASK';
  task_id number := 0;
  num_found number;
  sts_name varchar2(256) := 'SQLACCESS3638195_sts';
  sts_cursor dbms_sqltune.sqlset_cursor;
BEGIN
  /* Create Task */
  dbms_advisor.create_task(DBMS_ADVISOR.SQLACCESS_ADVISOR,
                           task_id,
                           taskname,
                           task_desc,
                           task_or_template);

  /* Reset Task */
  dbms_advisor.reset_task(taskname);

  /* Delete Previous STS Workload Task Link */
  select count(*)
  into   num_found
  from   user_advisor_sqla_wk_map
  where  task_name = taskname
  and    workload_name = sts_name;
  IF num_found > 0 THEN
    dbms_advisor.delete_sqlwkld_ref(taskname,sts_name,1);
  END IF;

  /* Delete Previous STS */

 /* select count(*)
  into   num_found
  from   user_advisor_sqlw_sum
  where  workload_name = sts_name;
  IF num_found > 0 THEN
    dbms_sqltune.delete_sqlset(sts_name);
  END IF; */

  /* Create STS */
  dbms_sqltune.create_sqlset(sts_name, 'Obtain workload from cursor cache');

  /* Select all statements in the cursor cache. */
  OPEN sts_cursor FOR
    SELECT VALUE(P)
    FROM TABLE(dbms_sqltune.select_cursor_cache) P;

  /* Load the statements into STS. */
  dbms_sqltune.load_sqlset(sts_name, sts_cursor);
  CLOSE sts_cursor;

  /* Link STS Workload to Task */
  dbms_advisor.add_sqlwkld_ref(taskname,sts_name,1);

  /* Set STS Workload Parameters */
  dbms_advisor.set_task_parameter(taskname,'VALID_ACTION_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'VALID_MODULE_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'SQL_LIMIT','25');
  dbms_advisor.set_task_parameter(taskname,'VALID_USERNAME_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'VALID_TABLE_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'INVALID_TABLE_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'INVALID_ACTION_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'INVALID_USERNAME_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'INVALID_MODULE_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'VALID_SQLSTRING_LIST',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'INVALID_SQLSTRING_LIST','"@!"');

  /* Set Task Parameters */
  dbms_advisor.set_task_parameter(taskname,'ANALYSIS_SCOPE','ALL');
  dbms_advisor.set_task_parameter(taskname,'RANKING_MEASURE','PRIORITY,OPTIMIZER_COST');
  dbms_advisor.set_task_parameter(taskname,'DEF_PARTITION_TABLESPACE',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'TIME_LIMIT',10000);
  dbms_advisor.set_task_parameter(taskname,'MODE','LIMITED');
  dbms_advisor.set_task_parameter(taskname,'STORAGE_CHANGE',DBMS_ADVISOR.ADVISOR_UNLIMITED);
  dbms_advisor.set_task_parameter(taskname,'DML_VOLATILITY','TRUE');
  dbms_advisor.set_task_parameter(taskname,'WORKLOAD_SCOPE','PARTIAL');
  dbms_advisor.set_task_parameter(taskname,'DEF_INDEX_TABLESPACE',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'DEF_INDEX_OWNER',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'DEF_MVIEW_TABLESPACE',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'DEF_MVIEW_OWNER',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'DEF_MVLOG_TABLESPACE',DBMS_ADVISOR.ADVISOR_UNUSED);
  dbms_advisor.set_task_parameter(taskname,'CREATION_COST','TRUE');
  dbms_advisor.set_task_parameter(taskname,'JOURNALING','4');
  dbms_advisor.set_task_parameter(taskname,'DAYS_TO_EXPIRE','30');

  /* Execute Task */
  dbms_advisor.execute_task(taskname);
END;
/

++++++++  Check recommendations ++++++++

SET LONG 100000
SET PAGESIZE 50000
SELECT DBMS_ADVISOR.get_task_script('SQLACCESS3638195') AS script FROM   dual;
SET PAGESIZE 24



Thursday 19 February 2015

SQL Performance Analyzer

Real Application Testing is used to test the impact of any change e.g upgrade, on the database. It has two components:

 1. Database Replay
 2. SQL Performance Analyzer

Database Replay is a tool for capturing the real workload in database and play it at will. But it captures the entire workload of the database.
SQL performance Analyzer is used to know the impact of any change e.g parameter change, on one or more SQLs.

Steps to use SQLPA:

1. Create SQLSET using DBMS_SQLTUNE.CREATE_SQLSET

2. Load SQLSET ( from Cursors,awr or sql traces) using DBMS_SQLTUNE.LOAD_SQLSET

3. Create SQLPA analysis task using DBMS_SQLPA.CREATE_ANALYSIS_TASK

4. Execute analysis task using DBMS_SQLPA.EXECUTE_ANALYSIS_TASK

5. Report analysis task using DBMS_SQLPA.REPORT_ANALYSIS_TASK

Wednesday 18 February 2015

Recovering voting disks/OCR in 11gr2

In case voting diskgroup is lost or corrupted we need to create new ASM diskgroup. But we cannot start ASM as voting disk is missing.

Here is how we can recover voting disk in this scenario:

 Stop crs on all the nodes
   
      # crsctl stop crs -f
   
 Start crs in exclusive mode on one of the nodes (node1)
   
     # crsctl start crs -excl  <- For 11201 RAC
   
     # crsctl start crs -excl -nocrs  <- For 11202 RAC
   
 If we don’t have the OCR DISK GROUP, then we need to create the disk group else move to restoring OCR DISK
   
 Check latest auto backup of OCR:
   
        $GRID_HOME/bin/ocrconfig –showbackup

  Restore the latest OCR backup(root user)

      # ocrconfig -restore $GRID_HOME/cdata/<hostname>/backup00.ocr


Recreate the Voting file (root user)

The Voting file needs to be initialized in the CRS disk group

      # crsctl replace votedisk +OCR_DG  <-- new or existing DG
   
      Above command will Re-create/move your voting disk in the specified ASM Disk Group, if you query the voting disk it will display your voting disk location in the DISK Group which has been specified.
   
     

Sunday 15 February 2015

RAC processes

LMON: Lock Monitor Process

LMON maintains GCS memory structures. It handles the abnormal termination of processes and instances. Reconfiguration of locks and resources when an instance joins or leaves the cluster are handled by LMON. Activities that occur during instance reconfigurations are tracked by it in its trace file. In versions 10gR2 and later, LMON is responsible for executing dynamic lock remastering every 10 minutes. In current versions, LMON is known as the Global Enqeueue Services monitor.

LMS: Lock Manager Server

LMS is the most active Oracle RAC background process; it can become very active, consuming significant amounts of CPU time. Oracle recommends that this process be allocated the needed CPU time by increasing its priority. Starting from 10g R2, Oracle has implemented a feature to ensure that the LMS process does not encounter CPU starvation. It is also known as the Global Cache Service (GCS) process in current versions.


Load Balancing Advisory and Connection Load Balancing in RAC

Oracle Net Services provides the ability to balance client connections across the instances in an Oracle RAC configuration. There are two types of load balancing that you can implement:

Client-side load balancing. Client-side load balancing balances the connection requests across the listeners. Set via LOAD_BALANCE parameter in the the net service name within a TNSNAMES.ORA file, it is just a random selection of the addresses in the address_list section of the net service name.

Server-side load balancing .With server-side load balancing, the listener directs a connection request to the best instance currently providing the service. Set via REMOTE_LISTENER in SPFILE, each instance registers with the TNS listeners running on all nodes within the cluster. By default this load balance is done on the instance on node. This can be changed to session based if required.

From 10g release 2 the service can be setup to use load balancing advisory. This mean connections can be routed using SERVICE TIME and THROUGHPUT.  Connection load balancing  means the goal of a service can be changed, to reflect the type of connections using the service.


Configuring the Load Balancing Advisory 

GOAL

None(0):When set to 0(NONE), this disables the ONS notifications to the Client as to the load of the various nodes.

Service Time(1)Attempts to direct work requests to instances according to response time. So if one node takes longer to do the same work, the client can be informed of this load difference, so it can now
direct further work to the node that is taking less time.Load balancing advisory data is based on elapsed time for work done in the service plus available bandwidth to the service.

Example to modify a service called ORCL and setup the service  to use SERVICE_TIME

EXECUTE DBMS_SERVICE.MODIFY_SERVICE (service_name => 'ORCL'  
, goal => DBMS_SERVICE.GOAL_SERVICE_TIME -  
, clb_goal => DBMS_SERVICE.CLB_GOAL_SHORT);

Throughput(2):Attempts to direct work requests according to throughput. The load balancing advisory is based on the rate that work is completed in the service plus available bandwidth to the service. Instead of figuring out how long something takes, it is the frequency this work occurs that is used.
So if node one is able to handle 10  transactions, while node two can handle 12, in the same amount of time, then the client will be told to go to node two. So even if node two will take longer to handle a specific job, it can handle more jobs at one time then node.

EXECUTE DBMS_SERVICE.MODIFY_SERVICE (service_name => 'ORCL' -
        , goal => DBMS_SERVICE.GOAL_THROUGHPUT -
       , clb_goal => DBMS_SERVICE.CLB_GOAL_LONG);
You can see the goal settings for a service in the data dictionary and in the DBA_SERVICES, V$SERVICES, and V$ACTIVE_SERVICES views

Configuring Connection Load Balancing

CLB_GOAL.

Long(2). Use the LONG connection load balancing method for applications that have long-lived connections. This is typical for connection pools and SQL*Forms sessions. It does not matter if GOAL is set or not for this condition as the point behind this setting is to balance based on  number of sessions. LONG is the default connection load balancing goal.

Example to modify service ORCL and set CLB_GOAL long

EXECUTE DBMS_SERVICE.MODIFY_SERVICE (service_name => 'ORCL'
        , clb_goal => DBMS_SERVICE.CLB_GOAL_LONG);
Short(1).Use the SHORT connection load balancing method for applications that have short-lived connections. The database uses first the GOAL setting to have PMON tell the Listener
which node to prefer

Example to modify service ORCL and set CLB_GOAL short

EXECUTE DBMS_SERVICE.MODIFY_SERVICE (service_name => 'ORCL'
, CLB_GOAL => DBMS_SERVICE.CLB_GOAL_SHORT);

Please note for 11.2 onwards for DBMS_SERVICE.MODIFY_SERVICE >> This procedure is deprecated in databases managed by Oracle Clusterware and Oracle Restart with Release 11.2. While the procedure remains available in the package, Oracle recommends using srvctl to manage services. This is because the service attributes are stored in CRS by srvctl, and overwrite those specified by DBMS_SERVICE. The DBMS_SERVICE procedures do not update the CRS attributes.

11gr2 RAC root.sh logs

Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /usr/orasys/11.2.0.4_CRS
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /usr/orasys/11.2.0.4_CRS/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'node1'
CRS-2676: Start of 'ora.mdnsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node1'
CRS-2676: Start of 'ora.gpnpd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node1'
CRS-2672: Attempting to start 'ora.gipcd' on 'node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'node1'
CRS-2676: Start of 'ora.diskmon' on 'node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'node1' succeeded

ASM created and started successfully.

Disk Group OCRDATA created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 1c2ea8c5a03b4fe6bf6b95eb3b128c61.
Successful addition of voting disk 6bf246f51e0a4fc5bfea31c8d9c6f9ea.
Successful addition of voting disk 17ab9993ffa64f62bfbef943867fad32.
Successfully replaced voting disk group with +OCRDATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c2ea8c5a03b4fe6bf6b95eb3b128c61 (ORCL:ORA_LRAC10_CRS_01) [OCRDATA]
 2. ONLINE   6bf246f51e0a4fc5bfea31c8d9c6f9ea (ORCL:ORA_LRAC10_CRS_02) [OCRDATA]
 3. ONLINE   17ab9993ffa64f62bfbef943867fad32 (ORCL:ORA_LRAC10_CRS_03) [OCRDATA]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.OCRDATA.dg' on 'node1'
CRS-2676: Start of 'ora.OCRDATA.dg' on 'node1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Apply PSU patch in Grid Infrastructures 11gr2

Before applying any patch it is best practice to go through the ReadMe of  the patch.

The patch installations differs based on following aspects of existing configuration. E.g
  • GI home is shared or non-shared
  • The Oracle RAC database home is shared or non-shared
  • The Oracle RAC database home software is on ACFS or non-ACFS file systems.
  • Patch all the Oracle RAC database and the GI homes together, or patch each home individually


So choose the most appropriate case that is suitable based on the existing configurations and patch intention.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Steps:

opatch lsinventory -detail -oh $GI_HOME
opatch lsinventory -detail -oh $ORACLE_HOME
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/ocm/bin/emocmrsp  ( scp ocmrf.rs to all nodes)

opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 13348650 -oh $GI_HOME

opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 13348650/custom/server/13348650

# $GI_HOME/OPatch/opatch auto ./patch -oh $GI_HOME -ocmrf ocm.rsp
# $ORACLE_HOME/OPatch/opatch auto ./patch -oh $ORACLE_HOME -ocmrf ocm.rsp


Now do same on other nodes.

Once done on all nodes, verify it:

opatch lsinventory -detail -oh $GI_HOME

From any one node:

SQL> @catbundle.sql psu apply

++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Detailed steps:


Prerequisites:

1.  OPatch utility version 11.2.0.1.8 or later. Install it as below if version is lower:

cd /usr/orasys/patches

229:+ASM1_node1:/usr/orasys/patches> ls -lrt

-rw-r--r-- 1 oracle dba   32510817 Aug 20 02:44 p6880880_112000_Linux-x86-64.zip
-rw-r--r-- 1 oracle dba 1156126720 Aug 20 02:44 p13348650_112030_Linux-x86-64.tar

unzip p6880880_112000_Linux-x86-64.zip -d /usr/orasys/11.2.0.3_CRS

$ORACLE_HOME/OPatch/opatch version <- should be 11.2.0.1.8 or later now

2. Create OCM configuration as follows:

/usr/orasys/11.2.0.3_CRS/OPatch/ocm/bin/emocmrsp     --- As Oracle user


Validation of Oracle Inventory:

Inventory should be consistent for oracle homes to be patches ( both GI and RDBMS homes)

%<ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

Unzip patch:
tar xvf p13348650_112030_Linux-x86-64.tar
Determine whether any currently installed one-off patches conflict with the PSU patch as follows:
In the unzipped directory as in 
 opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/13343438 –oh $GRID_HOME
opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/13343438/custom/server/13343438 –oh $DB_HOME

Note that Oracle proactively provides PSU one-off patches for common conflicts.
Use My Oracle Support Note 1061295.1 Patch Set Updates - One-off Patch Conflict Resolution to determine, for each conflicting patch, whether a conflict resolution patch is already available, and if you need to request a new conflict resolution patch or if the conflict may be ignored.

When all the one-off patches that you have requested are available at My Oracle Support, proceed with Patch installation.


Apply patch now:


Ask SEs to run below command as root user:

Note: it must be executed on each node in the cluster if the GI home or Oracle RAC database home is in Non-shared storage. The utility should not be run in parallel on the cluster nodes.
Depending on command line options specified, one invocation of Opatch can patch the GI home, one or more Oracle RAC database homes, or both GI and Oracle RAC database homes of the same Oracle release version. You can also roll back the patch with the same selectivity.


Add the directory containing the opatch to the $PATH environment variable. For example:
export PATH=$PATH:<GI_HOME>/OPatch
To patch GI home and all Oracle RAC database homes of the same version:
#opatch auto <UNZIPPED_PATCH_LOCATION> -ocmrf <ocm response file>


Output of above cmd would be as follows:

[root@node1 patches]# /usr/orasys/11.2.0.3_CRS/OPatch/opatch auto ./p13348650_112030_Linux-x86-64 -oh /usr/orasys/11.2.0.3_CRS -ocmrf ocm.rsp

Executing /usr/bin/perl /usr/orasys/11.2.0.3_CRS/OPatch/crs/patch112.pl -patchdir . -patchn p13348650_112030_Linux-x86-64 -oh /usr/orasys/11.2.0.3_CRS -ocmrf ocm.rsp -paramfile /usr/orasys/11.2.0.3_CRS/crs/install/crsconfig_params
opatch auto log file location is /usr/orasys/11.2.0.3_CRS/OPatch/crs/../../cfgtoollogs/opatchauto2015-08-12_04-42-49.log
Detected Oracle Clusterware install
Using configuration parameter file: /usr/orasys/11.2.0.3_CRS/crs/install/crsconfig_params
Successfully unlock /usr/orasys/11.2.0.3_CRS
patch ./p13348650_112030_Linux-x86-64/13348650  apply successful for home  /usr/orasys/11.2.0.3_CRS
patch ./p13348650_112030_Linux-x86-64/13343438  apply successful for home  /usr/orasys/11.2.0.3_CRS
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@node1 patches]#

- Load Modified SQL Files into the Database

SQL> @catbundle.sql psu apply


References: https://updates.oracle.com/Orion/Services/download?type=readme&aru=14279366

Waits events in RAC


There are four categories of waits events in RAC: 

1. Block-oriented:

gc current block 2-way
gc current block 3-way
gc cr block 2-way
gc cr block 3-way

2. Message-oriented:

gc current grant 2-way
gc cr grant 2-way

3. Contention-oriented:

    gc current block busy
    gc cr block busy
    gc buffer busy acquire/release

4. Load-oriented:

gc current block congested
gc cr block congested

The block-oriented wait event statistics indicate that a block was received as either the result of a 2-way or a 3-way message, that is, the block was sent from either the resource master requiring 1 message and 1 transfer, or was forwarded to a third node from which it was sent, requiring 2 messages and 1 block transfer.

The gc current block busy and gc cr block busy wait events indicate that the local instance that is making the request did not immediately receive a current or consistent read block. The term "busy" in these events' names indicates that the sending of the block was delayed on a remote instance. For example, a block cannot be shipped immediately if Oracle Database has not yet written the redo for the block's changes to a log file.

In comparison to "block busy" wait events, a gc buffer busy event indicates that Oracle Database cannot immediately grant access to data that is stored in the local buffer cache. This is because a global operation on the buffer is pending and the operation has not yet completed. In other words, the buffer is busy and all other processes that are attempting to access the local buffer must wait to complete.

The existence of gc buffer busy events also means that there is block contention that is resulting in multiple requests for access to the local block. Oracle Database must queue these requests. The length of time that Oracle Database needs to process the queue depends on the remaining service time for the block. The service time is affected by the processing time that any network latency adds, the processing time on the remote and local instances, and the length of the wait queue.

The average wait time and the total wait time should be considered when being alerted to performance issues where these particular waits have a high impact. Usually, either interconnect or load issues or SQL execution against a large shared working set can be found to be the root cause.

The message-oriented wait event statistics indicate that no block was received because it was not cached in any instance. Instead a global grant was given, enabling the requesting instance to read the block from disk or modify it.

If the time consumed by these events is high, then it may be assumed that the frequently used SQL causes a lot of disk I/O (in the event of the cr grant) or that the workload inserts a lot of data and needs to find and format new blocks frequently (in the event of the current grant).

The contention-oriented wait event statistics indicate that a block was received which was pinned by a session on another node, was deferred because a change had not yet been flushed to disk or because of high concurrency, and therefore could not be shipped immediately. A buffer may also be busy locally when a session has already initiated a cache fusion operation and is waiting for its completion when another session on the same node is trying to read or modify the same data. High service times for blocks exchanged in the global cache may exacerbate the contention, which can be caused by frequent concurrent read and write accesses to the same data.

The load-oriented wait events indicate that a delay in processing has occurred in the GCS, which is usually caused by high load, CPU saturation and would have to be solved by additional CPUs, load-balancing, off loading processing to different times or a new cluster node.For the events mentioned, the wait time encompasses the entire round trip from the time a session starts to wait after initiating a block request until the block arrives.

++++++++

gc cr request
The gc cr request wait event specifies the time it takes to retrieve the data from the remote cache. In Oracle 9i and prior, gc cr requestwas known as global cache cr request. High wait times for this wait event often are because of:
RAC Traffic Using Slow Connection – typically RAC traffic should use a high-speed interconnect to transfer data between instances, however, sometimes Oracle may not pick the correct connection and instead route traffic over the slower public network. This will significantly increase the amount of wait time for the gc cr request event. The oradebug command can be used to verify which network is being used for RAC traffic:
This will dump a trace file to the location specified by the user_dump_dest Oracle parameter containing information about the network and protocols being used for the RAC interconnect.
Inefficient Queries – poorly tuned queries will increase the amount of data blocks requested by an Oracle session. The more blocks requested typically means the more often a block will need to be read from a remote instance via the interconnect.
gc buffer busy acquire and gc buffer busy release
The gc buffer busy acquire and gc buffer busy release wait events specify the time the remote instance locally spends accessing the requested data block. In Oracle 11g you will see gc buffer busy acquire wait event when the global cache open request originated from the local instance and gc buffer busy release when the open request originated from a remote instance. In Oracle 10g these two wait events were represented in a single gc buffer busy wait, and in Oracle 9i and prior the “gc” was spelled out as “global cache” in the global cache buffer busy wait event. These wait events are all very similar to the buffer busy wait events in a single-instance database and are often the result of:
Hot Blocks – multiple sessions may be requesting a block that is either not in buffer cache or is in an incompatible mode. Deleting some of the hot rows and re-inserting them back into the table may alleviate the problem. Most of the time the rows will be placed into a different block and reduce contention on the block. The DBA may also need to adjust the pctfree and/or pctused parameters for the table to ensure the rows are placed into a different block.
Inefficient Queries – as with the gc cr request wait event, the more blocks requested from the buffer cache the more likelihood of a session having to wait for other sessions. Tuning queries to access fewer blocks will often result in less contention for the same block.


Saturday 14 February 2015

Changing network interface for interconnect (private IP)



Check existing settings:

To check nodes and status:

./olsnodes -s

To check n/w interface on a node:

/sbin/ifconfig -a


Add new interface  in cluster (new interface must be operational)

oifcfg setif -global if_name/subnet:cluster_interconnect

oifcfg setif -global eth1/10.10.0.0:couster_interconnect


Delete existing interface:

oifcfg delif -global if_name/subnet

oifcfg delif -global eth1/192.168.0.0

Verify new configuration

oifcfg getif

Stop Oracle Clusterware on all nodes as root user:

# crsctl crs stop

When Oracle Clusterware stops on all nodes, deconfigure the deleted network interface in the operating system using the ifconfig command

# ifconfig eth0 down

Update n/w config files in OS


Restart Oracle Clusterware by running the following command on each node in the cluster as the root user:

# crsctl start crs


If you use the CLUSTER_INTERCONNECTS initialization parameter, then you must update it to reflect the changes.

Changing VIPs in RAC


1. Stop all services running on existing VIPs

srvctl stop service -d grid -s serviceA,ServiceB -n node1

$ srvctl config vip -n existingvip
VIP exists.:stbdp03
VIP exists.: /node1-existingvip/192.168.2.20/255.255.255.0/eth0


Stop the existing VIP resource using the srvctl stop vip command:

$ srvctl stop vip -n node1


Verify that the VIP resource is no longer running by running the ifconfig -a


Make any changes necessary to the /etc/hosts files on all nodes and make any necessary DNS changes to associate the new IP address with the old host name.

To use a different subnet or NIC for the default network before you change any VIP resource, you must use the srvctl modify network -S subnet/netmask/interface command as root to change the network resource, where subnet is the new subnet address, netmask is the new netmask, and interface is the new interface.

Modify the node applications and provide the new VIP address using the following srvctl modify nodeapps syntax:

$ srvctl modify nodeapps -n node1 -A new_vip_address

srvctl modify nodeapps -n node -A 192.168.2.125/255.255.255.0/eth0

Start the node VIP by running the srvctl start vip command:

$ srvctl start vip -n node_name

Repeat the steps for each node in the cluster.

Verify new config for all nodes:

$ cluvfy comp nodecon -n all -verbose