OCI - Configure site-to-site VPN with Meraki MX100

 Writing this blog on how we can configure OCI site-to-site VPN with Meraki MX100.

In this blog we have 2 steps. One to configure on OCI side another to setup on Meraki side.

1. OCI side configuration:

Pre-Requisite: I already have VCN created with private subnet and DRG attached to VCN.

a.  First configure your on-premises device (the customer-premises equipment, or CPE) at your end of the Site-to-Site VPN so traffic can flow between your on-premises network and virtual cloud network (VCN).

The CPE is a virtual representation of your customer-premises equipment, which is the actual router on-premises at your site at your end of the Site-to-Site VPN IPSec connection.

Click on Networking --> customer connectivity --> Customer-premises equipment

Click on create CPE

You need a public IP address of CPE to create CPE.

Also need to select vendor from the list (Meraki is not in the list so choose Other) and click create CPE.

b. Now Click on networking --> Site-to-site VPN

Click on Create IPSec connection

here we need select CPE we created in previous step, DRG and on-prem CIDR block.

I choose static routing here, You can change even after tunnel creation if need dynamic.

Oracle ask us to create 2 tunnel for redundancy purpose.

click on create IPSec connection and it will take couple of minutes to provisioned and we can see its UP.

you can also see 2 Oracle VPN IP address, called VPN head ends.

2. Next is setup on Meraki side.

we have to share these 2 VPN IP address and pre-shared keys with customer to setup on CPE side.

Here are the settings on Meraki side.

After this you will see both tunnel Up IPSec status Up in OCI

something like this

So the path should be like this from OCI VM to on-prem network.

DataPump Import job failing on multitenant environment due to ORA-01001: invalid cursor

Recently I was doing the database migration using expdp/impdp from on-prem non-CDB to ExaCC PDB. 

While doing import into PDB I face below issue.

[oracle@hostname dump]$ impdp parfile=imp_full.par

Import: Release 19.0.0.0.0 - Production on Sat Jun 3 23:32:20 2023
Version 19.17.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.

Connected to: Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
ORA-39006: internal error
ORA-39065: unexpected master process exception in DISPATCH
ORA-01001: invalid cursor
ORA-06512: at "SYS.DBMS_METADATA_UTIL", line 3405
ORA-06512: at "SYS.KUPM$MCP", line 2635

ORA-39097: Data Pump job encountered unexpected error -1001

After checking on MOS I found the document that suggest the cause and solution of this problem.

CAUSE: DataPump utility may get corrupted and we need to recreate DataPump utility to overcome internal corruption.

Solution as per MOS: 

DataPump Job Fails To Start Due To ORA-01001: invalid cursor (Doc ID 2354428.1) and

How To Reload Datapump Utility EXPDP/IMPDP (Doc ID 430221.1)

On Multitenant Environment, for Oracle versions 12.1, 12.2, 18c, 19c, 21c, and 23c:

On Multitenant Environment, for Oracle versions 12.1, 12.2, 18c, 19c, 21c, and 23c:

-- 1. To rebuild the DataPump packages with the following steps.
Under the ORACLE_HOME, execute:
cd rdbms/admin

-- run the dpload.sql in the CDB with all of the PDBs open, as described in "How to execute sql scripts in Multitenant environment (catcon.pl)" NOTE 1932340.1

Syntax:
$ORACLE_HOME/perl/bin/perl catcon.pl -d /u01/oracle -l "logging directory" -b "prefix for logfile of dpload for each PDB" dpload.sql

 
Where:
-l - directory to use for spool log files; for example, /tmp
-b - base name for log and spool file names; for example, dp_rebuild_log_base_name

For example, at OS prompt

cd $ORACLE_HOME/rdbms/admin

$ORACLE_HOME/perl/bin/perl catcon.pl -d /u01/oracle -l /tmp -b dp_rebuild_log_base_name dpload.sql

We tried above solution as per MOS doc but that did not work and failing with error.

catcon::sureunlink: unlink(/exaccnfs/ZDM/PDB/DUMP/log/ppb3.log) succeeded after 1 attempt(s)

catcon::sureunlink: verify that the file really no longer exists

catcon::sureunlink: confirmed that /exaccnfs/ZDM/PDB/DUMP/log/ppb3.log no longer exists after 1 attempts

catcon::delete_idle_logs: Deleting log file /exaccnfs/ZDM/PDB/DUMP/log/ppb4.log because SQL*Plus process for which
  it was created did no work

So the solution worked for us is when we run the dpload.sql script from sqlplus. 

NOTE: In CDB environment, all PDBS must be in open state.

@$ORACLE_HOME/rdbms/admin/dpload.sql;

After this Import started without issue.

Wallet_type shows "UNKNOWN" for all PDBs after changing wallet password on ExaCC

Recently I changed the sys and wallet password of CDB using this nice blog from my friend Kwame. 

Modify sys and wallet password on EXACC

After changing sys and wallet password, when I checked wallet status for all containers, I found wallet_type is UNKNOWN for all PDB's.

SQL> set linesize 200
SQL> col wallet_dir for a50
SQL> col status for a21
SQL> select con_id, status, wrl_parameter wallet_dir, wallet_type from v$encryption_wallet;

    CON_ID STATUS                WALLET_DIR                                         WALLET_TYPE
---------- --------------------- -------------------------------------------------- --------------------
         1 OPEN                  /var/opt/oracle/dbaas_acfs/CDB1/wallet_root/tde/   AUTOLOGIN
         2 OPEN                                                                     AUTOLOGIN
         5 OPEN                                                                     UNKNOWN
         6 OPEN                                                                     UNKNOWN
         7 OPEN                                                                     UNKNOWN

The Wallet type should be AUTOLOGIN for all PDBS's.

CAUSE: This is happening because we used mkstore utility to modify wallet credentials.

Solution: 1. Disable auto-login by renaming the .sso file

[oracle@hostname tde]$ mv cwallet.sso cwallet.sso.bkp

2. Close auto-login keystore

SQL> alter system set wallet close;

System altered.

SQL> select con_id, status, wrl_parameter wallet_dir, wallet_type from v$encryption_wallet;

    CON_ID STATUS                WALLET_DIR                                         WALLET_TYPE
---------- --------------------- -------------------------------------------------- --------------------
         1 CLOSED                /var/opt/oracle/dbaas_acfs/CDB1/wallet_root/tde/   UNKNOWN
         2 CLOSED                                                                   UNKNOWN
         5 CLOSED                                                                   UNKNOWN
         6 CLOSED                                                                   UNKNOWN
         7 CLOSED                                                                   UNKNOWN

3. Open password protected wallet

SQL> administer key management set keystore open identified by wallet_password;

keystore altered.

4. Re-enable autologin keystore

SQL> administer key management create auto_login keystore from keystore identified by wallet_password;

keystore altered.

5. Close password protected wallet. (Since we have recreated auto-login wallet back, we can close password-protected keystore to implicitly enable auto-login wallet.)

SQL> administer key management set keystore close identified by wallet_password;

keystore altered.

6. Now Verify if wallet type is autologin or not

SQL> select con_id, status, wrl_parameter wallet_dir, wallet_type from v$encryption_wallet;

    CON_ID STATUS                WALLET_DIR                                         WALLET_TYPE
---------- --------------------- -------------------------------------------------- --------------------
         1 OPEN                  /var/opt/oracle/dbaas_acfs/CDB1/wallet_root/tde/   AUTOLOGIN
         2 OPEN                                                                     AUTOLOGIN
         5 OPEN                                                                     AUTOLOGIN
         6 OPEN                                                                     AUTOLOGIN
         7 OPEN                                                                     AUTOLOGIN

And it's BACK! wallet_type shows AUTOLOGIN for all PDB's now.

"TDE Keystore password is not valid" when cloning a PDB after changing wallet password on EXACC

Recently I changed the sys and wallet password of CDB using this nice blog from my friend Kwame. 

Modify sys and wallet password on EXACC

But the difference this time is, I have one existing PDB inside this CDB, it's not the empty shell CDB for which above blog tested.

I tried the above blog and password change worked and everything looks great, but when we test to clone the existing PDB using local_clone, we received an error.. 

TDE keystore password is not valid

[oracle@test admin]$ dbaascli pdb local_clone --pdbname PDB_SEED_DM --target_pdbname DMTC --dbname CDBDB1
DBAAS CLI version 23.1.2.0.0
Executing command pdb local_clone --pdbname PDB_SEED_DM --target_pdbname DMTC --dbname CDBDB1
Job id: 668921cd-2762-4869-bafc-7e52c3aa91f3
Session log: /var/opt/oracle/log/CDBDB1/pdb/localClone/dbaastools_2023-06-01_04-37-56-PM_123807.log
Loading PILOT...
Session ID of the current execution is: 3446
Log file location: /var/opt/oracle/log/CDBDB1/pdb/localClone/pilot_2023-06-01_04-38-00-PM_125221
-----------------
Running Plugin_initialization job
Completed Plugin_initialization job
-----------------
Running Validate_input_params job
Completed Validate_input_params job
-----------------
Running Perform_dbca_prechecks job
Execution of Perform_dbca_prechecks failed
[FATAL] [DBAAS-60071] Operation has failed with following error message:
 [FATAL] [DBT-16031] TDE keystore password is not valid
   CAUSE: Unable to open the wallet using the provided TDE keystore credentials.
   ACTION: Specify the correct TDE keystore credentials..
   SUMMARY:
       - [DBAAS-60022] Command '/var/opt/oracle/dbaastools/dbaasca/bin/dbca -silent -oui_internal -createPluggableDatabase -sourceDB CDBDB1_MCTN -pdbName DMTC -pdbAdminUserName ADMIN -createPDBFrom PDB -sourcePDB PDB_SEED_DM -skipPasswordComplexityCheck ***** -checkOnly -jreLoc /usr/java/jdk1.8.0_361-amd64/jre ' has failed on nodes [localnode].
*** Executing jobs which need to be run always... ***
******** PLUGIN EXECUTION FAILED ********

But when I checked wallet passsword, everything looks fine to me.

Somehow exisitng PDB not syncronized with changed wallet password, so how to synchronized this PDB with changed wallet password.

There is no command to do that, so what i tried is below command..

NOTE: make sure you use same changed wallet password in old and new password ( we are not changing password here)

[oracle@test ~]$ dbaascli tde changePassword --pdbname PDB_SEED_DM --dbname CDBDB1
DBAAS CLI version 23.1.2.0.0
Executing command tde changePassword --pdbname PDB_SEED_DM --dbname CDBDB1
Job id: 31335ee2-01dc-4a86-872a-3e1a92248461
Session log: /var/opt/oracle/log/CDBDB1/tde/changePassword/dbaastools_2023-06-01_06-02-27-PM_349483.log
Enter Old keystore password:

Enter New keystore password:

Enter New keystore password (reconfirmation):

Loading PILOT...
Enter Old keystore password                                                                                                                                                                          ******************
Enter Old keystore password (reconfirmation):                                                                                                                                                      *******************
Enter New keystore password                                                                                                                                                                            ****************
Enter New keystore password (reconfirmation):                                                                                                                                                           *********************
Session ID of the current execution is: 3451
Log file location: /var/opt/oracle/log/CDBDB1/tde/changePassword/pilot_2023-06-01_06-02-39-PM_352160
-----------------
Running Plugin_initialization job
Completed Plugin_initialization job
-----------------
Running TDE_change_password_prechecks job
Completed TDE_change_password_prechecks job
-----------------
Running Extract_blob_in_standby job
Skipping. Job is detected as not applicable.
-----------------
Running Copy_blob_contents_in_standby job
Skipping. Job is detected as not applicable.
-----------------
Running TDE_change_password job
Completed TDE_change_password job
-----------------
Running Change_hsm_password job
Skipping. Job is detected as not applicable.
-----------------
Running Regenerate_auto_login_wallet job
Skipping. Job is detected as not applicable.
-----------------
Running Update_db_wallet job
Completed Update_db_wallet job
-----------------
Running Prepare_blob_for_standby_in_primary job
Skipping. Job is detected as not applicable.
dbaascli execution completed
[oracle@test ~]$  

After this , i tried the PDB clone again and it worked!

[oracle@test ~]$ dbaascli pdb local_clone --pdbname PDB_SEED_DM --target_pdbname DMTC --dbname CDBDB1
DBAAS CLI version 23.1.2.0.0
Executing command pdb local_clone --pdbname PDB_SEED_DM --target_pdbname DMTC --dbname CDBDB1
Job id: 76735c65-f5dc-4b9f-b46d-e82ce3ff72b0
Session log: /var/opt/oracle/log/CDBDB1/pdb/localClone/dbaastools_2023-06-01_06-04-01-PM_29543.log
Loading PILOT...
Session ID of the current execution is: 3452
Log file location: /var/opt/oracle/log/CDBDB1/pdb/localClone/pilot_2023-06-01_06-04-06-PM_31288
-----------------
Running Plugin_initialization job
Completed Plugin_initialization job
-----------------
Running Validate_input_params job
Completed Validate_input_params job
-----------------
Running Perform_dbca_prechecks job
Completed Perform_dbca_prechecks job
-----------------
Running PDB_creation job
Completed PDB_creation job
-----------------
Running Load_pdb_details job
Completed Load_pdb_details job
-----------------
Running Configure_pdb_service job
Completed Configure_pdb_service job
-----------------
Running Configure_tnsnames_ora job
Completed Configure_tnsnames_ora job
-----------------
Running Set_pdb_admin_user_profile job
Completed Set_pdb_admin_user_profile job
-----------------
Running Lock_pdb_admin_user job
Completed Lock_pdb_admin_user job
-----------------
Running Register_ocids job
Skipping. Job is detected as not applicable.
-----------------
Running Prepare_blob_for_standby_in_primary job
Skipping. Job is detected as not applicable.
-----------------
Running Generate_dbsystem_details job
Completed Generate_dbsystem_details job
dbaascli execution completed
[oracle@test ~]$