We have a very basic domain: 3 physical servers, 1 containing the admin server, 2 containing 2 managed servers each, running in a cluster. Nice picture to follow. 🙂

name of the servers: server01 (Admin) / server02 (MS1+MS2) / server03 (MS3+MS4)

Step 0 – The Weblogic Demo Certificates (the reason why)

If you ever checked your Weblogic production logs and found the Security Alert below, then your Weblogic domain has a huge gaping security hole. Even if you don’t want SSL communication for your webservices or applications. The internal (administrative) processes in your Weblogic domain still relies on the default demotrust and with this everyone can access your domain.


<Alert> <Security> <SERVER02> <rbx_dev_soa_ms01> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <BEA-090152> <Demo trusted CA certificate is being used in production mode: [
  Version: V3
  Subject: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4

  Key:  Sun RSA public key, 512 bits
  modulus: 95501928778692415440753....
  public exponent: 65537
  Validity: [From: Thu Mar 21 21:12:27 CET 2002,
               To: Tue Mar 22 21:12:27 CET 2022]
  Issuer: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  SerialNumber: [    33f10648 fcde0deb 4199921f d64537f4]

Certificate Extensions: 1
[1]: ObjectId: Criticality=true
KeyUsage [

  Algorithm: [MD5withRSA]
0000: 9D 26 4C 29 C8 91 C3 A7   06 C3 24 6F AE B4 F8 82  .&L)......$o....
0010: 80 4D AA CB 7C 79 46 84   81 C4 66 95 F4 1E D8 C4  .M...yF...f.....
0020: E9 B7 D9 7C E2 23 33 A4   B7 21 E0 AA 54 2B 4A FF  .....#3..!..T+J.
0030: CB 21 20 88 81 21 DB AC   90 54 D8 7D 79 63 23 3C  .! ..!...T..yc#<

] The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.>

Step 1 – generate the identity java keystore

We first generate a java keystores containing a private key for each physical server, since SSL hostname verifiation is based on machine names. We use “welcome1” as password here, but make sure to choose a complex password here, use it during these steps and then lock it safely away.

keytool.exe -genkey -keysize 2048 -keyalg RSA -alias server01 -keystore server01_identity.jks

Enter keystore password: welcome1
 Re-enter new password: welcome1
 What is your first and last name?
   [Unknown]:  server01
 What is the name of your organizational unit?
 What is the name of your organization?
   [Unknown]:  Rubix
 What is the name of your City or Locality?
 What is the name of your State or Province?
 What is the two-letter country code for this unit?
   [Unknown]:  NL
 Is CN=server01, OU=, O=Rubix, L=, ST=, C=NL correct?
   [no]:  yes

Step 2 – generate signing request

keytool.exe -certreq -alias server01 -file server01.csr -keystore server01_identity.jks</div>
Repeat step 1 + 2 for each physical server, replacing the -alias parameter, -keystore parameter and CN name with the correct server0* value.


Step 3 – the Certificate Authority (CA)

Now comes the tricky part that you probably need some outside forces. You will need the correct Certificate Authority to sign your certificate requests (the .CSR files you generated). You can create your own CA and self-sign them, but I would not recommend that unless you know what you are doing and understand the consequences. You could use an external Internet CA provider, but this becomes very costly and time consuming when you need such an external provider to sign every SSL enabled server in your landscape. The best situation for us would be if the current organization already has an internal CA provider, especially when the rootCA is trusted by the servers and machines in your landscape.
Let’s assume however that you receive the signed response, including the public keys of the rootCA and a intermediateCA. CA instances often use a intermediate CA to authorize certificates, so if they have “problems” they can withdraw this intermediate CA and don’t need to withdraw the whole RootCA making it useless.
So the situation should be that you know hold 3 PEM/DER files.


Step 4 – importing the CA response

Import the certificates in your keystore, starting with the rootCA, then the intermediateCA, then the specific server alias. Example for server01:
keytool -importcert -trustcacerts -alias rootca -file rootCA.cer -keystore server01_identity.jks

keytool -importcert -alias intermediateca -file intermediateCA.cer -keystore server01_identity.jks

keytool -importcert -alias server01 -file server01.cer -keystore server01_identity.jks
Repeat these steps for server02 and server03. So we know have 3 identity keystores for each server containing the signed private key and it’s certificate chain.


Step 5 – generate the trust java keystore

Now let’s generate a truststore for each server. Altough you can use the identical keystore for both identity and trust, it’s always a good practice to separate them. The identity keystone contains the private key and normally never changes during it’s lifetime, so you should secure it with a complex password. The trust store contains the certificates that you trust and as your middleware landscape grows, this file might grow and more administrators will need access to it.
Example generating the truststore for server01, repeat for 02 & 03.
keytool -importcert -trustcacerts -alias rootca -file rootCA.cer -keystore server01_trust.jks
keytool -importcert -alias intermediateca -file intermediateCA.cer -keystore server01_trust.jks

Step 6 – copy the java keystore files to each server

 Make sure that each server had the files stored on disk. We use the windows server example “c:oraclesecurity” in this blogpost. So server01 should hold:
– c:oraclesecurityserver01_identity.jks
– c:oraclesecurityserver01_trust.jks


Step 7 – configure the Node Manager for SSL

The nodemanger by default uses the demoIdentity and demoTrust keystores.
So on each server hosting a managed server (server02 and server03) we will need to change this.
Open the nodemanager.properties and add the following lines:
### SSL config
# CustomIdentityKeyStoreType=JKS
# CustomTrustKeyStoreType=JKS
Restart the NodeManager to activate the changes. The passwords in the properties file will automatically become encrypted.
Possible error:
For server rbx_dev_soa_ms03, the Node Manager associated with machine server03 is not reachable. All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.
Possible solution:
Your AdminServer can not communicate with a node manager. Make sure it’s running, check Environment -> Machines in the Weblogic console if the machine has state reachable, check if the address configured there, and the address in the nodemanager.properties and the CN-name used in SSL have the same value, check if Weblogic and Nodemanager use the same SSL configuration (trust+identity).


Step 8 – configure Weblogic for SSL

Log in the Weblogic console, in our example: http://server01:7001/console
Step 8.1 – configure Weblogic for SSL, enable SSL
Log in the Weblogic console (in our case http://server01:7001/console) and make sure that each server has the SSL port enabled. You can find the setting “SSL Listen Port Enabled” for each server under “Configuration -> General”. Enable it and make sure you use a correct and free port.

Step 8.2 – configure Weblogic for SSL, changing keystores

For each server (Admin + MS) we have to change the KeyStore configuration.
So for each server go to: Configuration -> Keystores
  • Keystores -> “Change” = “Custom Identity and Custom Trust”
  • Custom Identity Keystore = c:/Oracle/security/serverXX_identity.jks
  • Custom Identity Keystore Type = JKS
  • Custom Identity Keystore Passphrase = welcome1
  • Custom Trust keystore = c:/Oracle/security/serverXX_trust.jks
  • Custom Trust Keystore Type = JKS
  • Custom Trust Keystore Passphrase = welcome1

Step 8.3 – configure Weblogic for SSL, changing identity:

For each server (Admin + MS) we have to change the SSL configuration.
So for each server go to: Configuration -> SSL
  • Private Key Alias = serverXX
  • Private Key Passphrase = welcome1

Step 9 – Weblogic start scripts:

By default Weblogic uses the DemoTrust.jks truststore. Since we changed this in the console configuration you would hope this would be enough. However the SetDomainEnv.sh / SetDomainEnv.cmd script still contains a reference. So for each server:
  • Go to %domain_home%/bin/
  • Open the SetDomainEnv script
  • Search the argument: -Djavax.net.ssl.trustStore=
  • Make sure the value is not referring to the demo trust.jks but “c:/oracle/security/serverXX_trust.jks
When you are finished restart your whole Weblogic environment (Admin + MS).

Step 10 – Configure Enterprise Manager:

This took me some research and the Oracle Forums to get it working. Navigate to the enterprise manager console (/em)

Step 10.1 – Security Provider Configuration:

I’m actually not really sure if this step is still necessary in 11g, but hey .. why not be sure:
  • Click Weblogic -> right-click domain
  • Select Security -> Security Provider Configuration -> KeyStore Configure
  • Keystore Path = c:/oracle/security/server01_identity.jks

Step 10.2 – SOA Infra properties

  • Click SOA -> right-click soa-infra
  • Select SOA Administration -> Common Properties
  • Click More SOA Infra Advanced Properties
  • Keystore Location = c:/oracle/security/serverXX_identity.jks

Step 10.3 – Credential Mapping:
The next step is to configure the credential mapping keys

  • Click Weblogic -> right-click domain name
  • Select Security -> Credentials
  • Create a Map named “SOA”
  • Click Create a key
  • Key = KeyPassword
  • User Name = KeyPassword
  • Password = welcome1
  • Click Create Key:
  • Key = KeyStorePassword
  • User Name = KeyStorePassword
  • Password = welcome1
Possible error:

<Warning> <oracle.as.jmx.framework.MessageLocalizationHelper> <J2EE JMX-46041> <The resource for bundle "oracle.soa.management.config.bpel.mbeans.MessageBundle_en" with key "oracle.soa.BPELConfigBean.auditFlushByteThreshold" cannot be found.>
INFO: SSLSocketFactoryManagerImpl.getKeystoreLocation SOA Keystore location: c:/oracle/security/server04_identity.jks
INFO: SSLSocketFactoryManagerImpl.getKeystorePassword Obtained null or empty keystore password
INFO: SSLSocketFactoryManagerImpl.getKeyPassword Obtained null or empty key password
INFO: SSLSocketFactoryManagerImpl.getSSLSocketFactory Could not obtain keystore location or password

Possible solution:
You did not configure the SOA key mapping correctly as mentioned above in 9.3



Enabling SSL takes a lot of steps, and we didn’t even mentioned it for enabling SSL client identity in OSB (Service Key Provider) and communication with an LDAP over SSL. But there’s more to come I guess.
If you have any remarks please let me know in the comments. Hope it helps. 🙂