Understanding and repairing ASM disk layout using kfed

Kfed parameters

• aun - Allocation Unit (AU) number to read from. Default is AU0, or the very beginning of the ASM disk.
• aus – AU size. Default is 1048576 (1MB). Specify the aus when reading from a disk group with non-default AU size.
• blkn – block number to read. Default is block 0, or the very first block of the AU.
• dev – ASM disk or device name. Note that the keyword dev can be omitted, but the ASM disk name is mandatory.

Read ASM disk header block from  AU[0]
[root@grac41 Desktop]# kfed read  /dev/asm_test_1G_disk1 | egrep 'name|size|type'
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD<-- ASM disk header
kfdhdb.dskname:               TEST_0000 ; 0x028: length=9   <-- ASM disk name
kfdhdb.grpname:                    TEST ; 0x048: length=4   <-- ASM DG name
kfdhdb.fgname:                TEST_0000 ; 0x068: length=9   <-- ASM Failgroup
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200     <-- Disk sector size   
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000     <-- ASM block size
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000 <-- AU size : 1 Mbyte 
kfdhdb.dsksize:                    1023 ; 0x0c4: 0x000003ff <-- ASM disk size : 1 GByte  

Check ASM block types for the first 2 AUs 
AU[0] :
[root@grac41 Desktop]# kfed find /dev/asm_test_1G_disk1
Block 0 has type 1
Block 1 has type 2
Block 2 has type 3
Block 3 has type 3
Block 4 has type 3
Block 5 has type 3
Block 6 has type 3
Block 7 has type 3
Block 8 has type 3
Block 9 has type 3
Block 10 has type 3
..
Block 252 has type 3
Block 253 has type 3
Block 254 has type 3
Block 255 has type 3

AU[1] :
[root@grac41 Desktop]#  kfed find /dev/asm_test_1G_disk1 aun=1
Block 256 has type 17
Block 257 has type 17
Block 258 has type 13
Block 259 has type 18
Block 260 has type 13
..
Block 508 has type 13
Block 509 has type 13
Block 510 has type 1
Block 511 has type 19

Summary :
--> Disk header size is 512 bytes 
    AU size = 1Mbyte  --> AU block size = 4096 
    This translates to 1048576 / 4096 = 256 blocks to read an AU ( start with block 0 - 255 )
    Block 510 and block 0 storing an ASM disk header ( == type 1 )

Run the kfed command below if you interested in a certain ASM block type ( use output from kfed find to the type info )
[root@grac41 Desktop]#  kfed read  /dev/asm_test_1G_disk1 aun=1 blkn=255  | egrep 'type'
kfbh.type:                           19 ; 0x002: KFBTYP_HBEAT

Some ASM block types 
[root@grac41 Desktop]# kfed read  /dev/asm_test_1G_disk1 aun=0 blkn=0  | egrep 'type' 
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.type:                            2 ; 0x002: KFBTYP_FREESPC
kfbh.type:                            3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type:                            5 ; 0x002: KFBTYP_LISTHEAD
kfbh.type:                           13 ; 0x002: KFBTYP_PST_NONE
kfbh.type:                           18 ; 0x002: KFBTYP_PST_DTA
kfbh.type:                           19 ; 0x002: KFBTYP_HBEAT

Repair ASM disk header block in AU[0] with kfed repair

In ASM versions 11.1.0.7 and later, the ASM disk header block is backed up in the second last ASM metadata block in the allocation unit 1.
 
Verify ASM DISK Header block located in  AU[0] and AU[1]
AU[0] :
[root@grac41 Desktop]# kfed read  /dev/asm_test_1G_disk1 aun=0 blkn=0 | egrep 'name|size|type'
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dskname:               TEST_0000 ; 0x028: length=9
kfdhdb.grpname:                    TEST ; 0x048: length=4
kfdhdb.fgname:                TEST_0000 ; 0x068: length=9
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize:                    1023 ; 0x0c4: 0x000003ff

AU[1] : 
[root@grac41 Desktop]# kfed read  /dev/asm_test_1G_disk1 aun=1 blkn=254  | egrep 'name|size|type'
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dskname:               TEST_0000 ; 0x028: length=9
kfdhdb.grpname:                    TEST ; 0x048: length=4
kfdhdb.fgname:                TEST_0000 ; 0x068: length=9
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize:                    1023 ; 0x0c4: 0x000003ff

Erase Disk header block in first AU ( aun=0 blkn=0 )
# dd if=/dev/zero of=/dev/asm_test_1G_disk1  bs=4096 count=1

Verify ASM disk header
# kfed read /dev/asm_test_1G_disk1 aun=0 blkn=0
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
--> Corrupted ASM disk header detected in AU [0]

Repair disk header in AU[0] with kfed 
[grid@grac41 ASM]$ kfed repair  /dev/asm_test_1G_disk1 
[grid@grac41 ASM]$ kfed read /dev/asm_test_1G_disk1 aun=0 blkn=0
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dskname:               TEST_0000 ; 0x028: length=9
kfdhdb.grpname:                    TEST ; 0x048: length=4
kfdhdb.fgname:                TEST_0000 ; 0x068: length=9
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize:                    1023 ; 0x0c4: 0x000003ff
--> kfed repair worked - Disk header restored 

Can kfed repair the Disk header block stored in the 2.nd AU ?


Delete  Disk header block in AU[1]
First use dd to figure out whether we are getting the correct block
[grid@grac41 ASM]$  dd if=/dev/asm_test_1G_disk1 of=-  bs=4096 count=1 skip=510 ; strings block1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000464628 s, 8.8 MB/s
ORCLDISK
TEST_0000
TEST
TEST_0000
--> looks like an ASM disk header - go ahead and erase that block

[grid@grac41 ASM]$  dd if=/dev/zero of=/dev/asm_test_1G_disk1  bs=4096 count=1  seek=510 
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00644028 s, 636 kB/s

Verify ASM disk header block in AU[1]
[grid@grac41 ASM]$ kfed read /dev/asm_test_1G_disk1 aun=1 blkn=254
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
--> Corrupted ASM disk header detected

[grid@grac41 ASM]$ kfed repair  /dev/asm_test_1G_disk1 
KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]
--> kfed repair doesn' work 

Repair block with dd
grid@grac41 ASM]$ dd if=/dev/asm_test_1G_disk1  bs=4096  count=1 of=/dev/asm_test_1G_disk1  bs=4096 count=1  seek=510 
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.0306682 s, 134 kB/s
[grid@grac41 ASM]$ kfed read /dev/asm_test_1G_disk1 aun=0 blkn=0
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dskname:               TEST_0000 ; 0x028: length=9
kfdhdb.grpname:                    TEST ; 0x048: length=4
kfdhdb.fgname:                TEST_0000 ; 0x068: length=9
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize:                    1023 ; 0x0c4: 0x000003ff

# kfed read /dev/asm_test_1G_disk1 aun=1 blkn=254
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dskname:               TEST_0000 ; 0x028: length=9
kfdhdb.grpname:                    TEST ; 0x048: length=4
kfdhdb.fgname:                TEST_0000 ; 0x068: length=9
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize:                    1023 ; 0x0c4: 0x000003ff

Summary:
 to fix the backup block or the ASM disk header in AU 1 block you need to use dd

Reference

http://www.hhutzler.de/blog/using-kfeddd-to-repair-asm-disk-header-corruption/

Startup sequence of 11gR2 Clusterware

Short summary of the startup sequence: INIT spawns init.ohasd (with respawn) which in turn starts the OHASD process (Oracle High Availability Services Daemon).  This daemon spawns 4 processes.

Level 1: OHASD Spawns:

• cssdagent - Agent responsible for spawning CSSD.
• orarootagent - Agent responsible for managing all root owned ohasd resources.
• oraagent - Agent responsible for managing all oracle owned ohasd resources.
• cssdmonitor - Monitors CSSD and node health (along wth the cssdagent).

Level 2: OHASD rootagent spawns:

• CRSD - Primary daemon responsible for managing cluster resources.
• CTSSD - Cluster Time Synchronization Services Daemon
• Diskmon
• ACFS (ASM Cluster File System) Drivers 

Level 2: OHASD oraagent spawns:

• MDNSD - Used for DNS lookup
• GIPCD - Used for inter-process and inter-node communication
• GPNPD - Grid Plug & Play Profile Daemon
• EVMD - Event Monitor Daemon
• ASM - Resource for monitoring ASM instances

Level 3: CRSD spawns:

• orarootagent - Agent responsible for managing all root owned crsd resources.
• oraagent - Agent responsible for managing all oracle owned crsd resources.

Level 4: CRSD rootagent spawns:

• Network resource - To monitor the public network
• SCAN VIP(s) - Single Client Access Name Virtual IPs
• Node VIPs - One per node
• ACFS Registery - For mounting ASM Cluster File System
• GNS VIP (optional) - VIP for GNS

Level 4: CRSD oraagent spawns:

• ASM Resouce - ASM Instance(s) resource
• Diskgroup - Used for managing/monitoring ASM diskgroups.  
• DB Resource - Used for monitoring and managing the DB and instances
• SCAN Listener - Listener for single client access name, listening on SCAN VIP
• Listener - Node listener listening on the Node VIP
• Services - Used for monitoring and managing services
• ONS - Oracle Notification Service
• eONS - Enhanced Oracle Notification Service
• GSD - For 9i backward compatibility
• GNS (optional) - Grid Naming Service - Performs name resolution

Recovering voting disks/OCR in 11gr2

In case voting diskgroup is lost or corrupted we need to create new ASM diskgroup. But we cannot start ASM as voting disk is missing.

Here is how we can recover voting disk in this scenario:

 Stop crs on all the nodes
   
      # crsctl stop crs -f
   
 Start crs in exclusive mode on one of the nodes (node1)
   
     # crsctl start crs -excl  <- For 11201 RAC
   
     # crsctl start crs -excl -nocrs  <- For 11202 RAC
   
 If we don’t have the OCR DISK GROUP, then we need to create the disk group else move to restoring OCR DISK
   
 Check latest auto backup of OCR:
   
        $GRID_HOME/bin/ocrconfig –showbackup

  Restore the latest OCR backup(root user)

      # ocrconfig -restore $GRID_HOME/cdata/<hostname>/backup00.ocr


Recreate the Voting file (root user)

The Voting file needs to be initialized in the CRS disk group

      # crsctl replace votedisk +OCR_DG  <-- new or existing DG
   
      Above command will Re-create/move your voting disk in the specified ASM Disk Group, if you query the voting disk it will display your voting disk location in the DISK Group which has been specified.
   
     

RAC processes

LMON: Lock Monitor Process

LMON maintains GCS memory structures. It handles the abnormal termination of processes and instances. Reconfiguration of locks and resources when an instance joins or leaves the cluster are handled by LMON. Activities that occur during instance reconfigurations are tracked by it in its trace file. In versions 10gR2 and later, LMON is responsible for executing dynamic lock remastering every 10 minutes. In current versions, LMON is known as the Global Enqeueue Services monitor.

LMS: Lock Manager Server

LMS is the most active Oracle RAC background process; it can become very active, consuming significant amounts of CPU time. Oracle recommends that this process be allocated the needed CPU time by increasing its priority. Starting from 10g R2, Oracle has implemented a feature to ensure that the LMS process does not encounter CPU starvation. It is also known as the Global Cache Service (GCS) process in current versions.

Load Balancing Advisory and Connection Load Balancing in RAC

Oracle Net Services provides the ability to balance client connections across the instances in an Oracle RAC configuration. There are two types of load balancing that you can implement:

Client-side load balancing. Client-side load balancing balances the connection requests across the listeners. Set via LOAD_BALANCE parameter in the the net service name within a TNSNAMES.ORA file, it is just a random selection of the addresses in the address_list section of the net service name.

Server-side load balancing .With server-side load balancing, the listener directs a connection request to the best instance currently providing the service. Set via REMOTE_LISTENER in SPFILE, each instance registers with the TNS listeners running on all nodes within the cluster. By default this load balance is done on the instance on node. This can be changed to session based if required.

From 10g release 2 the service can be setup to use load balancing advisory. This mean connections can be routed using SERVICE TIME and THROUGHPUT.  Connection load balancing  means the goal of a service can be changed, to reflect the type of connections using the service.


Configuring the Load Balancing Advisory 

GOAL

None(0):When set to 0(NONE), this disables the ONS notifications to the Client as to the load of the various nodes.

Service Time(1)Attempts to direct work requests to instances according to response time. So if one node takes longer to do the same work, the client can be informed of this load difference, so it can now
direct further work to the node that is taking less time.Load balancing advisory data is based on elapsed time for work done in the service plus available bandwidth to the service.

Example to modify a service called ORCL and setup the service  to use SERVICE_TIME

EXECUTE DBMS_SERVICE.MODIFY_SERVICE (service_name => 'ORCL'  
, goal => DBMS_SERVICE.GOAL_SERVICE_TIME -  
, clb_goal => DBMS_SERVICE.CLB_GOAL_SHORT);

Throughput(2):Attempts to direct work requests according to throughput. The load balancing advisory is based on the rate that work is completed in the service plus available bandwidth to the service. Instead of figuring out how long something takes, it is the frequency this work occurs that is used.
So if node one is able to handle 10  transactions, while node two can handle 12, in the same amount of time, then the client will be told to go to node two. So even if node two will take longer to handle a specific job, it can handle more jobs at one time then node.

EXECUTE DBMS_SERVICE.MODIFY_SERVICE (service_name => 'ORCL' -
        , goal => DBMS_SERVICE.GOAL_THROUGHPUT -
       , clb_goal => DBMS_SERVICE.CLB_GOAL_LONG);
You can see the goal settings for a service in the data dictionary and in the DBA_SERVICES, V$SERVICES, and V$ACTIVE_SERVICES views

Configuring Connection Load Balancing

CLB_GOAL.

Long(2). Use the LONG connection load balancing method for applications that have long-lived connections. This is typical for connection pools and SQL*Forms sessions. It does not matter if GOAL is set or not for this condition as the point behind this setting is to balance based on  number of sessions. LONG is the default connection load balancing goal.

Example to modify service ORCL and set CLB_GOAL long

EXECUTE DBMS_SERVICE.MODIFY_SERVICE (service_name => 'ORCL'
        , clb_goal => DBMS_SERVICE.CLB_GOAL_LONG);
Short(1).Use the SHORT connection load balancing method for applications that have short-lived connections. The database uses first the GOAL setting to have PMON tell the Listener
which node to prefer

Example to modify service ORCL and set CLB_GOAL short

EXECUTE DBMS_SERVICE.MODIFY_SERVICE (service_name => 'ORCL'
, CLB_GOAL => DBMS_SERVICE.CLB_GOAL_SHORT);

Please note for 11.2 onwards for DBMS_SERVICE.MODIFY_SERVICE >> This procedure is deprecated in databases managed by Oracle Clusterware and Oracle Restart with Release 11.2. While the procedure remains available in the package, Oracle recommends using srvctl to manage services. This is because the service attributes are stored in CRS by srvctl, and overwrite those specified by DBMS_SERVICE. The DBMS_SERVICE procedures do not update the CRS attributes.

11gr2 RAC root.sh logs

Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /usr/orasys/11.2.0.4_CRS
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /usr/orasys/11.2.0.4_CRS/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'node1'
CRS-2676: Start of 'ora.mdnsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node1'
CRS-2676: Start of 'ora.gpnpd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node1'
CRS-2672: Attempting to start 'ora.gipcd' on 'node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'node1'
CRS-2676: Start of 'ora.diskmon' on 'node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'node1' succeeded

ASM created and started successfully.

Disk Group OCRDATA created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 1c2ea8c5a03b4fe6bf6b95eb3b128c61.
Successful addition of voting disk 6bf246f51e0a4fc5bfea31c8d9c6f9ea.
Successful addition of voting disk 17ab9993ffa64f62bfbef943867fad32.
Successfully replaced voting disk group with +OCRDATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1c2ea8c5a03b4fe6bf6b95eb3b128c61 (ORCL:ORA_LRAC10_CRS_01) [OCRDATA]
 2. ONLINE   6bf246f51e0a4fc5bfea31c8d9c6f9ea (ORCL:ORA_LRAC10_CRS_02) [OCRDATA]
 3. ONLINE   17ab9993ffa64f62bfbef943867fad32 (ORCL:ORA_LRAC10_CRS_03) [OCRDATA]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.OCRDATA.dg' on 'node1'
CRS-2676: Start of 'ora.OCRDATA.dg' on 'node1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Apply PSU patch in Grid Infrastructures 11gr2

Before applying any patch it is best practice to go through the ReadMe of  the patch.

The patch installations differs based on following aspects of existing configuration. E.g

• GI home is shared or non-shared
• The Oracle RAC database home is shared or non-shared
• The Oracle RAC database home software is on ACFS or non-ACFS file systems.
• Patch all the Oracle RAC database and the GI homes together, or patch each home individually

So choose the most appropriate case that is suitable based on the existing configurations and patch intention.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Steps:

opatch lsinventory -detail -oh $GI_HOME
opatch lsinventory -detail -oh $ORACLE_HOME
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/ocm/bin/emocmrsp  ( scp ocmrf.rs to all nodes)

opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 13348650 -oh $GI_HOME

opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 13348650/custom/server/13348650

# $GI_HOME/OPatch/opatch auto ./patch -oh $GI_HOME -ocmrf ocm.rsp
# $ORACLE_HOME/OPatch/opatch auto ./patch -oh $ORACLE_HOME -ocmrf ocm.rsp


Now do same on other nodes.

Once done on all nodes, verify it:

opatch lsinventory -detail -oh $GI_HOME

From any one node:

SQL> @catbundle.sql psu apply

++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Detailed steps:

Prerequisites:

1.  OPatch utility version 11.2.0.1.8 or later. Install it as below if version is lower:

cd /usr/orasys/patches

229:+ASM1_node1:/usr/orasys/patches> ls -lrt

-rw-r--r-- 1 oracle dba   32510817 Aug 20 02:44 p6880880_112000_Linux-x86-64.zip
-rw-r--r-- 1 oracle dba 1156126720 Aug 20 02:44 p13348650_112030_Linux-x86-64.tar

unzip p6880880_112000_Linux-x86-64.zip -d /usr/orasys/11.2.0.3_CRS

$ORACLE_HOME/OPatch/opatch version <- should be 11.2.0.1.8 or later now

2. Create OCM configuration as follows:

/usr/orasys/11.2.0.3_CRS/OPatch/ocm/bin/emocmrsp     --- As Oracle user

Validation of Oracle Inventory:

Inventory should be consistent for oracle homes to be patches ( both GI and RDBMS homes)

%<ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

Unzip patch:



tar xvf p13348650_112030_Linux-x86-64.tar

Determine whether any currently installed one-off patches conflict with the PSU patch as follows:




In the unzipped directory as in 

 opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/13343438 –oh $GRID_HOME

opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/13343438/custom/server/13343438 –oh $DB_HOME

Note that Oracle proactively provides PSU one-off patches for common conflicts.

Use My Oracle Support Note 1061295.1 Patch Set Updates - One-off Patch Conflict Resolution to determine, for each conflicting patch, whether a conflict resolution patch is already available, and if you need to request a new conflict resolution patch or if the conflict may be ignored.

When all the one-off patches that you have requested are available at My Oracle Support, proceed with Patch installation.

Apply patch now:


Ask SEs to run below command as root user:

Note: it must be executed on each node in the cluster if the GI home or Oracle RAC database home is in Non-shared storage. The utility should not be run in parallel on the cluster nodes.
Depending on command line options specified, one invocation of Opatch can patch the GI home, one or more Oracle RAC database homes, or both GI and Oracle RAC database homes of the same Oracle release version. You can also roll back the patch with the same selectivity.

Add the directory containing the opatch to the $PATH environment variable. For example:

export PATH=$PATH:<GI_HOME>/OPatch

To patch GI home and all Oracle RAC database homes of the same version:

#opatch auto <UNZIPPED_PATCH_LOCATION> -ocmrf <ocm response file>

Output of above cmd would be as follows:

[root@node1 patches]# /usr/orasys/11.2.0.3_CRS/OPatch/opatch auto ./p13348650_112030_Linux-x86-64 -oh /usr/orasys/11.2.0.3_CRS -ocmrf ocm.rsp

Executing /usr/bin/perl /usr/orasys/11.2.0.3_CRS/OPatch/crs/patch112.pl -patchdir . -patchn p13348650_112030_Linux-x86-64 -oh /usr/orasys/11.2.0.3_CRS -ocmrf ocm.rsp -paramfile /usr/orasys/11.2.0.3_CRS/crs/install/crsconfig_params
opatch auto log file location is /usr/orasys/11.2.0.3_CRS/OPatch/crs/../../cfgtoollogs/opatchauto2015-08-12_04-42-49.log
Detected Oracle Clusterware install
Using configuration parameter file: /usr/orasys/11.2.0.3_CRS/crs/install/crsconfig_params
Successfully unlock /usr/orasys/11.2.0.3_CRS
patch ./p13348650_112030_Linux-x86-64/13348650  apply successful for home  /usr/orasys/11.2.0.3_CRS
patch ./p13348650_112030_Linux-x86-64/13343438  apply successful for home  /usr/orasys/11.2.0.3_CRS
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@node1 patches]#

- Load Modified SQL Files into the Database

SQL> @catbundle.sql psu apply


References: https://updates.oracle.com/Orion/Services/download?type=readme&aru=14279366

Waits events in RAC

There are four categories of waits events in RAC: 

1. Block-oriented:

gc current block 2-way

gc current block 3-way

gc cr block 2-way

gc cr block 3-way

2. Message-oriented:

gc current grant 2-way

gc cr grant 2-way

3. Contention-oriented:

    gc current block busy

    gc cr block busy

    gc buffer busy acquire/release

4. Load-oriented:

gc current block congested

gc cr block congested

The block-oriented wait event statistics indicate that a block was received as either the result of a 2-way or a 3-way message, that is, the block was sent from either the resource master requiring 1 message and 1 transfer, or was forwarded to a third node from which it was sent, requiring 2 messages and 1 block transfer.

The gc current block busy and gc cr block busy wait events indicate that the local instance that is making the request did not immediately receive a current or consistent read block. The term "busy" in these events' names indicates that the sending of the block was delayed on a remote instance. For example, a block cannot be shipped immediately if Oracle Database has not yet written the redo for the block's changes to a log file.

In comparison to "block busy" wait events, a gc buffer busy event indicates that Oracle Database cannot immediately grant access to data that is stored in the local buffer cache. This is because a global operation on the buffer is pending and the operation has not yet completed. In other words, the buffer is busy and all other processes that are attempting to access the local buffer must wait to complete.

The existence of gc buffer busy events also means that there is block contention that is resulting in multiple requests for access to the local block. Oracle Database must queue these requests. The length of time that Oracle Database needs to process the queue depends on the remaining service time for the block. The service time is affected by the processing time that any network latency adds, the processing time on the remote and local instances, and the length of the wait queue.

The average wait time and the total wait time should be considered when being alerted to performance issues where these particular waits have a high impact. Usually, either interconnect or load issues or SQL execution against a large shared working set can be found to be the root cause.

The message-oriented wait event statistics indicate that no block was received because it was not cached in any instance. Instead a global grant was given, enabling the requesting instance to read the block from disk or modify it.

If the time consumed by these events is high, then it may be assumed that the frequently used SQL causes a lot of disk I/O (in the event of the cr grant) or that the workload inserts a lot of data and needs to find and format new blocks frequently (in the event of the current grant).

The contention-oriented wait event statistics indicate that a block was received which was pinned by a session on another node, was deferred because a change had not yet been flushed to disk or because of high concurrency, and therefore could not be shipped immediately. A buffer may also be busy locally when a session has already initiated a cache fusion operation and is waiting for its completion when another session on the same node is trying to read or modify the same data. High service times for blocks exchanged in the global cache may exacerbate the contention, which can be caused by frequent concurrent read and write accesses to the same data.

The load-oriented wait events indicate that a delay in processing has occurred in the GCS, which is usually caused by high load, CPU saturation and would have to be solved by additional CPUs, load-balancing, off loading processing to different times or a new cluster node.For the events mentioned, the wait time encompasses the entire round trip from the time a session starts to wait after initiating a block request until the block arrives.

++++++++

gc cr request

The gc cr request wait event specifies the time it takes to retrieve the data from the remote cache. In Oracle 9i and prior, gc cr requestwas known as global cache cr request. High wait times for this wait event often are because of:

RAC Traffic Using Slow Connection – typically RAC traffic should use a high-speed interconnect to transfer data between instances, however, sometimes Oracle may not pick the correct connection and instead route traffic over the slower public network. This will significantly increase the amount of wait time for the gc cr request event. The oradebug command can be used to verify which network is being used for RAC traffic:

1 2	SQL> oradebug setmypid SQL> oradebug ipc

This will dump a trace file to the location specified by the user_dump_dest Oracle parameter containing information about the network and protocols being used for the RAC interconnect.

Inefficient Queries – poorly tuned queries will increase the amount of data blocks requested by an Oracle session. The more blocks requested typically means the more often a block will need to be read from a remote instance via the interconnect.

gc buffer busy acquire and gc buffer busy release

The gc buffer busy acquire and gc buffer busy release wait events specify the time the remote instance locally spends accessing the requested data block. In Oracle 11g you will see gc buffer busy acquire wait event when the global cache open request originated from the local instance and gc buffer busy release when the open request originated from a remote instance. In Oracle 10g these two wait events were represented in a single gc buffer busy wait, and in Oracle 9i and prior the “gc” was spelled out as “global cache” in the global cache buffer busy wait event. These wait events are all very similar to the buffer busy wait events in a single-instance database and are often the result of:

Hot Blocks – multiple sessions may be requesting a block that is either not in buffer cache or is in an incompatible mode. Deleting some of the hot rows and re-inserting them back into the table may alleviate the problem. Most of the time the rows will be placed into a different block and reduce contention on the block. The DBA may also need to adjust the pctfree and/or pctused parameters for the table to ensure the rows are placed into a different block.

Inefficient Queries – as with the gc cr request wait event, the more blocks requested from the buffer cache the more likelihood of a session having to wait for other sessions. Tuning queries to access fewer blocks will often result in less contention for the same block.

Changing network interface for interconnect (private IP)

Check existing settings:

To check nodes and status:

./olsnodes -s

To check n/w interface on a node:

/sbin/ifconfig -a


Add new interface  in cluster (new interface must be operational)

oifcfg setif -global if_name/subnet:cluster_interconnect

oifcfg setif -global eth1/10.10.0.0:couster_interconnect


Delete existing interface:

oifcfg delif -global if_name/subnet

oifcfg delif -global eth1/192.168.0.0

Verify new configuration

oifcfg getif

Stop Oracle Clusterware on all nodes as root user:

# crsctl crs stop

When Oracle Clusterware stops on all nodes, deconfigure the deleted network interface in the operating system using the ifconfig command

# ifconfig eth0 down

Update n/w config files in OS


Restart Oracle Clusterware by running the following command on each node in the cluster as the root user:

# crsctl start crs


If you use the CLUSTER_INTERCONNECTS initialization parameter, then you must update it to reflect the changes.

Changing VIPs in RAC

1. Stop all services running on existing VIPs

srvctl stop service -d grid -s serviceA,ServiceB -n node1

$ srvctl config vip -n existingvip
VIP exists.:stbdp03
VIP exists.: /node1-existingvip/192.168.2.20/255.255.255.0/eth0


Stop the existing VIP resource using the srvctl stop vip command:

$ srvctl stop vip -n node1


Verify that the VIP resource is no longer running by running the ifconfig -a


Make any changes necessary to the /etc/hosts files on all nodes and make any necessary DNS changes to associate the new IP address with the old host name.

To use a different subnet or NIC for the default network before you change any VIP resource, you must use the srvctl modify network -S subnet/netmask/interface command as root to change the network resource, where subnet is the new subnet address, netmask is the new netmask, and interface is the new interface.

Modify the node applications and provide the new VIP address using the following srvctl modify nodeapps syntax:

$ srvctl modify nodeapps -n node1 -A new_vip_address

srvctl modify nodeapps -n node -A 192.168.2.125/255.255.255.0/eth0

Start the node VIP by running the srvctl start vip command:

$ srvctl start vip -n node_name

Repeat the steps for each node in the cluster.

Verify new config for all nodes:

$ cluvfy comp nodecon -n all -verbose