PingAM 8.0.0

Migrate to file-based configuration

Prior to AM 8.0, all information (configuration and data) was stored in LDAP directories. AM 8 can read its configuration from JSON files instead of directory servers.

File-based configuration (FBC) is best-suited to a DevOps-style deployment, with the associated tools and practices of that approach.

Static FBC data is written to configuration files in the file system and checked into a source control system, such as Git.

AM instances are created as Docker images, with the FBC incorporated into the image.

Kubernetes deployment using file-based configuration.

You can insert variables into these configuration files before you check them into source control. The variables are substituted with the appropriate values at runtime when you start the Docker container. Using variables lets you reuse the same base configuration files for multiple instances, and different staging environments. For example, development, QA, or pre-production, which are then promoted to production.

In an existing deployment, you can migrate your configuration to FBC after you’ve upgraded the configuration to AM 8.

Before you start, read How AM reads FBC to understand the file-based configuration layers and their order of precedence.

These topics assume you have already upgraded to AM 8 using the upgrade wizard. If you upgraded using, amupgrade and Amster, first deploy AM 8 and import the upgraded amster configuration. Then continue with the migration to FBC.

Migrating to FBC involves the following high-level steps:

  1. Separate dynamic data from configuration data

  2. Export existing configuration data

  3. Migrate exported configuration to FBC

  4. Provide configuration files to AM

  5. Start AM in FBC mode

Separate dynamic data from configuration data

Most deployments have one or more LDAP directories to store static configuration and data (identities, CTS tokens, policies, agents, and applications). If you store configuration and data in a single LDAP directory, you must separate the dynamic data first.

This section assumes you have stored identities in a separate datastore and focuses on CTS tokens, policies, agents, and applications. Export the data from your existing DS server and import it into one or more DS servers specifically for that data.

You can use your existing directory to store your policies and applications, to avoid the overhead of setting up and managing additional DS instances during the migration. In this case, overwrite the contents of the directory with only the dynamic data. If you do use the existing instance, backup directory so that you can restore it if issues occur during the migration.

To assist with the export, this table shows the default backend IDs and base DNs for AM data stored in DS directories. The table assumes you installed DS with the appropriate setup profiles. Adjust the values to reflect how you configured your DS server to store this data.

There is no DS setup profile for UMA-related data. This table therefore shows the values suggested in Prepare external UMA datastores.
Data type Backend ID Base DN

CTS tokens

amCts

ou=famrecords,ou=openam-session,ou=tokens

Policies, applications and agents

cfgStore

ou=services,ou=am-config

UMA-related data

umaRsStore

dc=uma-resources,dc=example,dc=com

  1. Set up at least one external PingDS instance for your exported data, following these instructions:

  2. Use the export-ldif command on your current DS server instance to export the policies, applications, agents, CTS tokens and any UMA-related data from your existing configuration store.

    • This command exports the CTS tokens to an LDIF file named output-cts.ldif:

      $ /path/to/opendj/bin/export-ldif \
  --hostname ds.example.com \
  --port 4444 \
  --bindDn uid=admin \
  --bindPassword str0ngAdm1nPa55word \
  --backendId amCts \
  --ldifFile output-cts.ldif \
  --trustAll
Export task ExportTask-16a9dce5-ceb1-41f0-a94d-154246ba7393 scheduled to start immediately
…​
Export task ExportTask-16a9dce5-ceb1-41f0-a94d-154246ba7393 has been successfully completed

    • This command exports UMA-related data to an LDIF file named output-uma.ldif:

      $ /path/to/opendj/bin/export-ldif \
  --hostname ds.example.com \
  --port 4444 \
  --bindDn uid=admin \
  --bindPassword str0ngAdm1nPa55word \
  --backendId umaRsStore \
  --ldifFile output-uma.ldif \
  --trustAll
Export task ExportTask-16a9dce5-ceb1-41f0-a94d-154246ba7393 scheduled to start immediately
…​
Export task ExportTask-16a9dce5-ceb1-41f0-a94d-154246ba7393 has been successfully completed

    • Follow the steps in Migrate policy or application data to DS to export the policy and application data.

    You can find more information about this command in export-ldif.

  3. Import the dynamic data into the new DS server or servers you’ve installed for this purpose.

    • This command imports the CTS tokens into a DS server instance at ds-cts.example.com:

      $ /path/to/opendj/bin/import-ldif \
  --hostname ds-cts.example.com \
  --port 4444 \
  --bindDn uid=admin \
  --bindPassword str0ngAdm1nPa55word \
  --backendId amCts \
  --ldifFile output-cts.ldif \
  --trustAll
Import task ImportTask-2f93d32f-0599-4215-8d3e-ecb77164d64b scheduled to start immediately
…​
Import task ImportTask-2f93d32f-0599-4215-8d3e-ecb77164d64b has been successfully completed

    • This command imports the UMA-related data into a DS server instance at ds-uma.example.com:

      $ /path/to/opendj/bin/import-ldif \
  --hostname ds-uma.example.com \
  --port 4444 \
  --bindDn uid=admin \
  --bindPassword str0ngAdm1nPa55word \
  --backendId umaRsStore \
  --ldifFile output-uma.ldif \
  --trustAll
Import task ImportTask-2f93d32f-0599-4215-8d3e-ecb77164d64b scheduled to start immediately
…​
Import task ImportTask-2f93d32f-0599-4215-8d3e-ecb77164d64b has been successfully completed

    • Follow the steps in Migrate policy or application data to DS to import the policy and application data to the DS server you’ve set up for that purpose.

    You can find more information about this command in import-ldif.

  4. On your AM configuration server, delete the dynamic data that you have exported.

    Although you don’t have to do this, it does result in a cleaner migration of static configuration.

    This command deletes the policies and applications, stored by default under ou=services,ou=am-config.

    $ /path/to/opendj/bin/ldapdelete \
  --deleteSubtree \
  --hostname ds.example.com \
  --port 4444 \
  --bindDn uid=admin \
  --bindPassword str0ngAdm1nPa55word \
  --trustAll \
  "ou=services,ou=am-config"
# DELETE operation successful for DN ou=services,ou=am-config

Export existing configuration data

The migration utility consumes an LDIF export of the existing configuration.

Use the export-ldif command to export the data, supplying the following parameters:

  • The backend ID of the configuration store, cfgStore by default.

  • The base DN of the branch containing the configuration, ou=am-config by default.

  • An LDIF file to store the exported configuration.

The following example, run on the DS server, assumes the default parameters when DS is installed for AM configuration:

$ /path/to/opendj/bin/export-ldif \
--backendId cfgStore \
--includeBranch ou=am-config \
--ldifFile /path/to/export-config/exported-configuration.ldif \
--offline
…​
 category=BACKEND severity=INFO seq=34 msg=Exported 1528 entries and skipped 0 in 0 seconds (average rate 24254.0/sec)

The exported file contains the existing configuration under ou=am-config:

$ more /path/to/export-config/exported-configuration.ldif
dn: ou=am-config
objectClass: top
objectClass: untypedObject
ou: am-config
…​

Migrate exported configuration to FBC

The migration utility is an enhanced version of the AM amupgrade command, available in the AM .zip file.

  1. If you don’t already have it, download AM-8.0.0.zip from the Backstage download site.

  2. Extract the contents of the .zip archive.

  3. In the extracted directory, locate and extract the Config-Upgrader-8.zip archive.

  4. To migrate the LDAP configuration, run the amupgrade command with the fileBasedMode argument:

    $ /path/to/amupgrade \
--fileBasedMode \
--ignoreNoRuleTracking \
--inputConfig input-directory \
--output output-directory \
/path/to/amupgrade/rules/fbc/migrate-to-fbc.groovy
    Migration command-line arguments
    --amsterVersion, -a

    The version of Amster that will be used to import the configuration (for example 8.0.0). This parameter is required unless the --fileBasedMode flag is provided.

    --baseDn, -d

    Lets you set a custom base DN.

    --clean

    Specifies that the migration should remove all existing files from the output location directory.

    Default: true

    --fileBasedMode, -m

    Specify that the configuration to be upgraded is file-based configuration (not Amster-exported configuration).

    Default: false

    --ignoreNoRuleTracking

    Don’t check for the existence of the Configuration Version Service. This lets you run the migration tool against individual files. However, it means that rules can be run multiple times against those files. Most rules don’t support this. Use with caution.

    Default: false

    --ignoreRequiredSecretMapping

    Prevents the upgrade from terminating due to missing required secret label mappings. Use with caution.

    Default: false

    -i, --inputConfig

    The directory containing the LDIF configuration files that you’re migrating.

    --log-output, -l

    The logging output file; logs at FINE by default and FINEST in verbose mode.

    --prettyArrays

    Whether to print each JSON array value on a new line.

    Default: false

    -o | --output directory

    The directory on the file system to which the migration utility writes the generated FBC.

    --secretsConfigFile, -f

    The properties file that contains settings necessary to migrate credentials to secrets. This file is required if your rules contain secrets upgrades. Read the sampleconfig.properties file for more details.

    --sectionsSource, -s

    A file that specifies the structure (section details) of the output configuration.

    Without this option, the migration utility generates the configuration properties as flat files.

    This file can be a section.properties file for a specific service or an AM.war file that includes the structure for all services.

    Example section.properties file for JSON audit handler 
    ########################################################################################################################
# Common handler section properties
########################################################################################################################
commonHandler=enabled
commonHandler=topics

########################################################################################################################
# Common handler plugin section properties
########################################################################################################################
commonHandlerPlugin=handlerFactory

########################################################################################################################
# JSON handler section properties
########################################################################################################################
jsonConfig=location
jsonConfig=elasticsearchCompatible
jsonConfig=rotationRetentionCheckInterval
jsonFileRotation=rotationEnabled
jsonFileRotation=rotationMaxFileSize
jsonFileRotation=rotationFilePrefix
jsonFileRotation=rotationFileSuffix
jsonFileRotation=rotationInterval
jsonFileRotation=rotationTimes
jsonFileRetention=retentionMaxNumberOfHistoryFiles
jsonFileRetention=retentionMaxDiskSpaceToUse
jsonFileRetention=retentionMinFreeSpaceRequired
jsonBuffering=bufferingMaxSize
jsonBuffering=bufferingWriteInterval
    rules-file-path

    The path to the upgrade rules file.

    Although you’re not using this command to upgrade to a new version, the amupgrade command requires an upgrade rules file.

    Use the noop.groovy file (located in (/path/to/amupgrade/rules/fbc/noop.groovy) to migrate the configuration without applying any rules.

    --version, -V

    Prints the version of this configuration upgrader tool and exits.

    --verbose, -v

    Logs verbose output; outputs at FINE level to the console and FINEST level to files.

    For example:

    $ /path/to/amupgrade \
--fileBasedMode \
--ignoreNoRuleTracking \
--inputConfig /path/to/export-config/ \
--output /path/to/fbc/config/ \
/path/to/amupgrade/rules/fbc/migrate-to-fbc.groovy

    In this example, the new JSON configuration files are generated in /path/to/fbc/config/.

    A sample alpha realm configuration might look like the following:

    $ more /path/to/am/config/services/global/realms/root-alpha.json
{
  "metadata": {
    "realm": "/alpha",
    "entityType": "",
    "entityId": "L2FscGhh",
    "uid": "o=alpha,ou=services,ou=am-config",
    "sunServiceID": null,
    "objectClass": [
      "sunRealmService",
      "top"
    ],
    "pathParams": {},
    "ou": []
  },
  "data": {
    "_id": "root-alpha",
    "_type": {
      "_id": "",
      "name": "",
      "collection": false
    },
    "active": true,
    "aliases": [
      "alpha"
    ],
    "parentPath": "/",
    "name": "alpha"
  }
}

Provide configuration files to AM

  1. Stop AM or the container where it runs.

  2. Copy the FBC files to the location of the deployment layer of your AM 8.0 configuration, by default, /path/to/am/config/config/services.

    Learn more about the deployment layer in How AM reads FBC.

    AM reads FBC from its home directory. You’ll need access to this directory on your local host or in your Docker container.

    For local hosts, change to the $AM_HOME/config/services directory:

    $ cd $AM_HOME/config/services

  3. Copy the migrated configuration into the services directory.

    $ cp /path/to/fbc/config/ $AM_HOME/config/services

    If the directory contains default configuration, delete it before copying in your migrated configuration.

Start AM in FBC mode

When you’ve migrated your configuration to JSON files, you can point your upgraded AM instances to the FBC, and to the new policy and application datastores.

These instructions assume a local AM server running in an Apache Tomcat container. Adjust the instructions to suit your environment.

  1. Set the following Java variable to true:

    com.sun.identity.sm.sms_object_filebased_enabled=true

  2. Pass that variable to AM in some way. For example, for Tomcat you can set the variable in the CATALINA_OPTS options:

    export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.identity.sm.sms_object_filebased_enabled=true"

  3. Restart AM or the container where it runs.

  4. Start up and test your environment.

    Check that AM has the configuration you expect. You can also make changes to your configuration through the AM admin UI and check that those changes are made in the JSON configuration files.

Troubleshoot your FBC migration

Use these sections to help you troubleshoot if you encounter the following errors during a migration to FBC:

Failed to import /etc/am/configuration/global/DataStoreService.json
Upgrade scenario

FBC import after an Amster export

Error

user-data: [main] ERROR org.forgerock.amster.org.forgerock.openam.sdk.http.DefaultErrorHandler - Unhandled client error: [Status: 400 Bad Request] user-data: Failed to import /etc/am/configuration/global/DataStoreService.json

Explanation

The order of imported files causes the import of DataStoreService.json to fail because the datastore instances that it references (for external application and policy stores) have not yet been created.

Solution

Import the global/DataStoreInstance path first, then import the remaining configuration.

Invalid SSO Token
Upgrade scenario

FBC import after an Amster export

Error

ERROR: Error updating properties for server null and tab sdk com.sun.identity.sm.SMSException: Invalid SSO Token: Invalid SSO Token

Explanation

Importing the CTS store properties when moving to an external CTS store fails for any subsequent import files because the CTS session is no longer valid.

Solution

Import the global/Servers path first, then import global/DefaultCtsDataStoreStoreProperties and finally import the remaining configuration.

Data validation failed for the attribute, Secret Label
Upgrade scenario

FBC import after an Amster export of configuration prior to 7.5.0

Error

Failed to import …​output/global/KeyStoreSecretStore/default-keystore/KeyStoreMappings/am.default.authentication.nodes.persistentcookie.encryption.json ERROR org.forgerock.openam.sdk.http.ClientErrorException: 400 Bad Request: Data validation failed for the attribute, Secret Label

Explanation

The secret mapping labels for persistent cookies changed in 7.5. As a result an attempt to import previous names fails.

Solution

Before performing the import:

  1. Remove any instances of global/KeyStoreSecretStore/…/am.default.authentication.modules.persistentcookie.encryption.json

  2. Remove any instances of global/KeyStoreSecretStore/…/am.default.authentication.modules.persistentcookie.signing.json

  3. Rename any instances of global/KeyStoreSecretStore/…/am.default.authentication.nodes.persistentcookie.encryption to am.authentication.nodes.persistentcookie.encryption. Modify the contents of each renamed file and perform the same search and replace.

  4. When the import is complete, optionally add encryption and signing secrets in the realms in which the PersistentCookieAuthModule is used. The new secret mapping format is:

    am.authentication.modules.persistentcookie.persistent-cookie-instance-name.encryption

    am.authentication.modules.persistentcookie.persistent-cookie-instance-name.signing

services/customusernamecollector': Trying to redefine version 0.0 for path
Upgrade scenario

FBC import after an Amster export

Error

ERROR o.f.a.CrestApiProducer - ApiDescription 'frapi:openam/realm-config:2.0//services/customusernamecollector': Trying to redefine version 0.0 for path ''

Explanation

The CustomUsernameCollectorAuth module isn’t recognized as an authentication module after the Amster import causing a failure to access the global service configuration.

Solution

Before importing the upgraded Amster configuration, make sure the 'com.example.custom.CustomUsernameCollectorAuth' is included in the list of authenticators in the global/Authentication.json file. Edit that file and add it as follows:

"authenticators" : [ "com.sun.identity.authentication.modules.ad.AD",
...
"org.forgerock.openam.authentication.modules.amster.Amster",
"org.forgerock.openam.authentication.modules.CustomUsernameCollectorAuth" ]
This example includes line breaks for legibility. In your file, the modules should all be on a single line.
UMA settings aren’t applied
Upgrade scenario

FBC import after an Amster export

Error

Could not create token in token data store: Operation failed: Result Code: Insufficient Access Rights

Explanation

Amster doesn’t export the default server settings for UMA. It only exports specific server settings, which inherit from the default server settings.

Solution

Before you import the global/Servers/01/UmaDataStoreProperties.json file:

  • If your deployment includes UMA configuration that is inherited from the server defaults (for example, the configuration for UMA to use external datastores), modify the inherited property to be false so that the specific server settings override the server defaults.

  • Alternatively, remove the specific server file so that it’s not imported and doesn’t override the server defaults.

Amster import fails for global/Monitoring.json file
Upgrade scenario

FBC import after an Amster export

Error

`ERROR o.f.o.c.s.SmsJsonConverter - Invalid attribute named snmpPort specified.

WARN o.f.o.c.r.s.SmsSingletonProvider - ::SmsSingletonProvider

InvalidAttributeException on Update org.forgerock.openam.core.sms.InvalidAttributeException: Invalid attribute specified.`

Explanation

Performing an Amster import from an upgraded previous version export fails for global/Monitoring.json because the snmpPort and snmpEnabled attributes aren’t recognized.

Solution

Before importing the Monitoring.json file, remove the snmpPort and snmpEnabled attributes and values.

Errors related to the keystore
Upgrade scenario

Server startup with FBC

Errors

ERROR: Unable to read private key password file ERROR: mapPk2Cert.JKSKeyProvider: java.io.FileNotFoundException ERROR: EventService.restartPSearches() Unable to start listener class com.sun.identity.sm.ldap.LDAPEventManager for data store DataStoreId…​ [CONTINUED]Unable to retrieve a connection

Explanation

These errors can be thrown when AM is unable to find the required keystores.

Solution

Make sure your secret stores are configured correctly after the migration.