Database Migration Procedure

azanium, the migration software is a Python package, which will be used for running all migration commands. All commands are all required to be invoked from the same working directory, and from the same host machine.

The sections below will need to be repeated for each migration run.

Software Setup

Install via the URL: $AZANIUM_WHEEL_URL. This URL can be obtained by copying the link with the .whl extension, listed under the section “Assets” on latest release page.

python3 -m pip install --user --upgrade pip
pip3 install --user "$AZANIUM_WHEEL_URL"

Verify installation has succeeded this by using the –help command:

azanium --help

Configuration

For each migration run, the software needs to be configured with the FTP URL pointing to the data release:

azanium configure $FTP_URL $RELEASE_TAG

Enabling Slack notifications

Note

The configuration of the slack URL need only be done once, and will persist across migration runs.

Migration steps can optionally broadcast messages to a WormBase slack channel to inform interested parties about the progress of the migration.

To enable slack notifications to the #db-migration-events channel in the WormBase slack, specify the $WEBHOOK_URL as the value for –slack-url option to the configure command.

The value for $WEBHOOK_URL is available from the WormBase slack management console for the WormBase organisation.

Important

To obtain the Webhook URL using the following instructions, you must be logged in as an administrator to the WormBase Slack

To find the Webhook URL:
  1. Visit the Slack API Apps page (must be logged in as a manager)
  2. Click the active “azanium” application listed under “Your Apps”
  3. Click “Incoming Webhooks” under “Features” (left side-menu)
  4. In the listing of Webhook URLs, click the top-most (latest) Webhook URL, listed against the #db-migration-events channel.

An example of configuring the Slack Webhook URL in conjunction with the $FTP_URL required to specify the release:

azanium configure $FTP_URL --slack-url=$WEBHOOK_URL

Commands

The following steps below will migrate the ACeDB database to Datomic.

The end result will be a Datomic database, compressed as an archive on the host the migration is performed upon.

The location of the file should be:
/wormbase/datomic-db-backups/<RELEASE>.tar.xz

  1. Connect via ssh to the machine where you’ll run the migration commands

    Use either a screen or tmux session, e.g:

    tmux new-session -s azanium-commands

    Clean up after any previous migration:

    azanium clean-previous-state

  2. Install software

    azanium install

  3. Run the main migration

    azanium migrate

    This command will execute all the steps required to perform the migration:

    1. Extract all .ace files from the ACeDB database for the current release.
    2. Compress all .ace files
    3. Create the Datomic database
    4. Convert .ace files to EDN logs
    5. Sort all EDN logs by timestamp
    6. Import the EDN logs into the Datomic database
    7. Run a QA report on the database
    8. Backup the Datomic database

  4. Run the homology migration

    Ensure there is enough memory to perform this step. The easiest way to ensure this is to reboot the instance before running the command for this step.

    azanium migrate-homol
    1. Create a homology database
    2. Backup the homology database.
  1. Notify watchers of completion of the process.

  1. Notify watchers of the slack channel that the migration has completed.

    If slack integration was configured, you can use (e.g):

    azanium notify \
    "Migration of ACeDB WS254 to Datomic complete! :fireworks:"

    Otherwise, write a message manually to the #db-migration-events slack channel.

Resulting Products

The followings files are created by the migration:

Datomic Database:

/wormbase/datomic-db-backup/*/$WS_RELEASE.tar.xz

QA Report

/wormbase/$WS_RELEASE-report.csv

Log file:

/wormbase/logs/azanium.log

Other Resources

Datomic transactor logs directory:

/wormbase/datomic_free/log

circus log file (circus is the hypervisor for running the transactor):

/wormbase/circus-datomic-transactor.log

Other commands

The following may be useful when manual intervention is required.

Reset the migration to a step (prompts):

azanium reset-to-step

Manually restart the transactor:

circusctl restart datomic-transactor