.. include:: /includes.rst.txt .. comments - headings # with overline, for parts * with overline, for chapters = for sections - for subsections ^ for subsubsections " for paragraphs * for H5 + for H6 .. index:: Backup .. index:: Restore .. _edg_backup_and_restore: Backup and Restore ====================== Administrators can back up and restore an TQ Data Foundation workspace either by using the built-in Backup and Restore Utility or by manually backing up and restoring the workspace files. These two methods are described in more detail in the sections below. The Backup and Restore Utility can back up Data Foundation while the system is online and operational; while the manual backup process requires Data Foundation to be shut down while its files are backed up. Both methods require the Data Foundation web application to be restarted when being restored from a backup. Although these two methods can both be used to back up and restore the same Data Foundation server at different times, their artifacts are not interchangeable. That is, backups created by the Backup and Restore Utility cannot be used by the manual restore process, and vice versa. Administrators can also back up and restore Explorer using these same options. Backup and Restore Utility -------------------------- Administrators can use the Backup and Restore Utility to back up and restore the Data Foundation workspace and its collections. Please test this utility in a development environment to get familiar with the process before using it in a production environment. This utility should not replace your organization's existing backup strategy; it should be used to supplement and enhance the existing strategy. Important notes ^^^^^^^^^^^^^^^ * When restoring a workspace, the workspace performing the restore must match the workspace that originally generated the backup in two ways: * The two workspaces must be configured to use the same database type (e.g. Shared TDB or Data Platform) * The two workspaces must have the same *major* and *minor* version numbers; but the *patch* version numbers can differ. Please do not upgrade to a new version of Data Foundation prior to restoring from a backup, this will not work. Upgrade after the restore. * Each backup is created as a zip file. The zip file can be either downloaded to the user's local machine or stored as an object in an `Amazon S3 `_ bucket. See :ref:`ExternalSystemIntegrationManagementAdminPage` and :ref:`S3ConfigurationSection`. * Everything in the workspace is backed up and restored, with the exception of the read-only Data Foundation system files. * During the restore process, *all* existing TDB databases in the workspace will be deleted and replaced by only those TDB databases present in the backup file. * Data Platform-backed workspaces can be backed up and restored using this utility, but the restore process requires manual intervention. See the discussion below. * Once a restore is in progress, do not navigate away from the Backup/Restore page or shut down the server. It may take a while to perform the restore, so allow it to finish. If an error occurs, it may be necessary to restart the web server. Utility Backup ^^^^^^^^^^^^^^ Administrators can use the Backup and Restore Utility to back up the Data Foundation workspace via either the Backup/Restore page or an API call; and either method can be used to download a backup file or store the file in an Amazon S3 bucket. The backup process is straightforward: An administrator requests a backup and the utility generates a zip file containing the requested backup. Backup/Restore Page """"""""""""""""""" Administrators can use the Backup/Restore page to generate a backup using the Data Foundation UI. **Download** To download the backup file, leave the backup-related Amazon S3 settings unconfigured and press the **Create Local Backup** button. The backup file will be created and downloaded like any other browser download. **Amazon S3 Bucket** To store the backup file in an Amazon S3 bucket, configure the backup-related Amazon S3 settings. (See :ref:`ExternalSystemIntegrationManagementAdminPage` and :ref:`S3ConfigurationSection`.) If the appropriate Amazon S3 settings are configured, the Backup/Restore page will display the name of the target Amazon S3 bucket. If the settings are not completely configured, the Backup/Restore page will display links to the appropriate Data Foundation pages for configuring those settings. Once the settings are configured, press the **Create Backup** button. The backup file will be created and uploaded to the configured Amazon S3 bucket with a time-stamped object name that will look something like this: `TopBraid-EDG-Studio-Backup-20220615T195844Z.zip`. To verify the checksum of a downloaded backup zip file, execute one of the following commands, substituting the appropriate filename, and compare the result to the checksum displayed alongside the file on the Server Administration Backup/Restore page: Windows Command Prompt: .. code-block:: bash certutil -hashfile SHA256 Linux shell: .. code-block:: bash sha256sum API """ Administrators can use the Backup and Restore API to generate a backup using any HTTP client. This allows backups to be created using scripts that can be executed automatically by scheduled jobs and the like. More details about the Backup and Restore API can be found in the Data Foundation online OpenAPI documentation. **Download** To download a newly-generated backup file, use an HTTP `GET` request similar to this `curl` command: .. code-block:: bash curl --request GET \ http://localhost/tbl/backup \ --header "Accept: application/zip" \ --header "Authorization: Basic xxxxxxxxxxxxxx" \ --output backup.zip **Amazon S3 Bucket** To store a newly-generated backup file in an Amazon S3 bucket, use an HTTP `POST` request similar to this `curl` command: .. code-block:: bash curl --request POST \ http://localhost/tbl/s3-backups \ --header "Accept: application/json" \ --header "Authorization: Basic xxxxxxxxxxxxxx" Utility Restore ^^^^^^^^^^^^^^^ Administrators can use the Backup and Restore Utility to restore a previously backed up Data Foundation workspace via either the Backup/Restore page or an API call; and either method can be used to upload the backup file to be used for the restore or fetch the file from an Amazon S3 bucket. To restore an Data Foundation workspace, an administrator initiates the restore with a specified backup file, either an uploaded file or a file stored in an Amazon S3 bucket. This backup file is staged on the Data Foundation server and, in most cases, a restart of the Data Foundation web application is automatically initiated. Once Data Foundation is restarted, it will wipe clean the existing workspace and replace it with the contents of the staged backup file. When Data Foundation has completed its initialization, it will have been restored to the state it was in when the backup file was created. In some cases, initiating a restore will *not* automatically restart the Data Foundation web application, and the entire web server must be manually restarted: * **EDG Studio:** EDG Studio does not support restarting the web application. The entire web server must be manually restarted. * **Windows server:** Data Foundation cannot be restarted when the web server is running on Windows. The entire web server must be manually restarted. * **Data Platform:** The restoration of a Data Foundation workspace running on Data Platform requires some manual intervention; so it cannot be automatically restarted. See the discussion below. .. note:: When initiating a restore with the Backup and Restore Utility, since the user-specified backup file must be staged on the Data Foundation server, there must be enough disk space available to the server to hold the backup file. If there is not enough disk space, the restore will fail. Backup/Restore Page """"""""""""""""""" Administrators can use the Backup/Restore page to initiate a restore using the Data Foundation UI. **Upload** To upload a backup zip file for restore, press the **Choose File** button and select the desired file from the local workstation; then press the **Restore** button. The backup file will be uploaded and staged on the Data Foundation server and a system restart initiated. **Amazon S3 Bucket** To use a backup zip file stored in an Amazon S3 bucket for restore, configure the backup-related Amazon S3 settings. Once the appropriate Amazon S3 settings are configured, the Backup/Restore page will display a list of the objects in the configured Amazon S3 bucket. Press the **Restore** button next to the desired file in the list. That backup file will be downloaded from Amazon S3 and staged on the Data Foundation server and a system restart initiated. API """ Administrators can use the Backup and Restore API to initiate a restore using any HTTP client. This allows restores to be initiated using scripts that can be executed automatically by scheduled jobs and the like. More details about the Backup and Restore API can be found in the Data Foundation online OpenAPI documentation. **Upload** To upload a backup file for restore and initiate a system restart, use an HTTP `POST` request similar to this `curl` command: .. code-block:: bash curl --request POST \ http://localhost/tbl/backup \ --header "Accept: */*" \ --header "Content-Type: multipart/form-data" \ --header "Authorization: Basic xxxxxxxxxxxxxx" \ --include \ --form "file=@TopBraid-EDG-Studio-Backup-20220621T041100Z.zip;type=application/zip" **Amazon S3 Bucket** To use a backup zip file stored in an Amazon S3 bucket for restore and initiate a system restart, use an HTTP `POST` request similar to this `curl` command: .. code-block:: bash curl --request POST \ http://localhost/tbl/s3-backups/TopBraid-EDG-Studio-Backup-20220621T041100Z.zip/backup \ --header "Accept: application/json" \ --header "Authorization: Basic xxxxxxxxxxxxxx" To list the backup zips file stored in an Amazon S3 bucket, use an HTTP `GET` request similar to this `curl` command: .. code-block:: bash curl --request GET \ http://localhost/tbl/s3-backups \ --header "Accept: application/json" \ --header "Authorization: Basic xxxxxxxxxxxxxx" The `key`\ s of the objects in the returned list can be used in the path of the `POST` request above to indicate which backup file is to be used in the restore. .. _backup_and_restore_utility_data_platform: Data Platform """"""""""""" Administrators can use either the Backup and Restore Utility or the Backup and Restore API to *initiate* a restore of a set of clustered Data Platform Data Foundation workspaces; but this will only *stage* the backup-related files on the Data Foundation web server from which the restore was initiated. It will not restart any of the clustered Data Foundation workspaces, because Data Platform restores must be completed manually. When a Data Platform workspace is backed up using the Backup Utility or API, the produced zip file contains two nested zip files: * `backup.zip` contains the files from the Data Foundation node that originally performed the backup. These files will be the same across all the Data Foundation nodes. * `dp-server-patch-logs.zip` contains the Data Coordinator patch logs, which will be used to restore the Data Coordinator server. Use one of the current Data Foundation nodes (or a freshly-installed Data Foundation node paired with a freshly-installed Data Coordinator server) to initiate a restore, which will stage the necessary backup files; then follow these steps: #. Shut down all the Data Foundation nodes; while leaving the Data Coordinator server running. #. Restart the Data Foundation node that initiated the restore, automatically triggering its restoration from the locally staged `backup.zip` file. #. Once this Data Foundation node is successfully restored, shut it down again. (At this point, the Data Coordinator server will also have been restored.) #. Replace the *other* Data Foundation nodes' `edg/workspace` directory trees with the *restored* node's `edg/workspace` directory tree. #. Restart all the Data Foundation nodes. .. seealso:: For additional details, see :ref:`data_platform`. Manual Backup and Restore ------------------------- Administrators can manually back up and restore an Data Foundation workspace's files. The steps necessary depend on the workspace's configured datastore: * TDB (Shared or Separate) * Data Platform Manual Backup ^^^^^^^^^^^^^ TDB """ Steps to back up an Data Foundation workspace that is configured to use Separate or Shared TDB: #. Shut down the Data Foundation web server. #. Back up the entire Data Foundation workspace directory tree, `edg/workspace`; e.g. by adding it to a zip or tar. This backup will include the workspace's TDB databases. #. Store this backup as appropriate. #. Once the backup is complete, restart the Data Foundation web server. Data Platform """"""""""""" Steps to back up a cluster of Data Foundation workspaces that are configured to use Data Platform: #. Shut down all the Data Foundation nodes. #. Shut down the Data Coordinator server. #. Back up the entire Data Foundation workspace directory tree, `edg/workspace`, of one Data Foundation node; e.g. by adding it to a zip or tar. #. Back up the Data Coordinator server's `--base` (patch logs) directory; e.g. by adding it to a zip or tar. #. Store these two backups together as appropriate. #. Once the backup is complete, restart the Data Coordinator server. #. Restart all the Data Foundation nodes. Manual Restore ^^^^^^^^^^^^^^ TDB """ Steps to restore an Data Foundation workspace that is configured to use Separate or Shared TDB: #. Shut down the Data Foundation web server. #. Replace the entire Data Foundation workspace directory tree, `edg/workspace`, with the contents of a TDB workspace backup. #. Restart the Data Foundation web server. .. _manual_backup_and_restore_data_platform: Data Platform """"""""""""" Steps to restore a cluster of Data Foundation workspaces that are configured to use Data Platform: #. Shut down all the Data Foundation nodes. #. Shut down the Data Coordinator server. #. Replace the entire Data Foundation workspace directory tree, `edg/workspace`, of each Data Foundation node with the contents of a Data Platform node workspace backup. #. Replace the Data Coordinator server's `--base` (patch logs) directory with the contents of the corresponding Data Coordinator backup. #. Restart the Data Coordinator server. #. Restart all the Data Foundation nodes.