When installing or expanding an SAP HANA system to be a scale out (distributed) system some configuration considerations need to be taken into account.
In the event of a host failure in a distributed system, a standby host will take over the persistence of the failing host. This is done through the use of the SAP HANA Storage Connector API.
Fiber Channel Protocol
Any volumes connected to multiple hosts in a scale out deployment must be connected using a Host Group.
In order for an SAP HANA scale out system to be installed using FCP the global.ini file must first be configured:
- Identify the WWIDN of the block storage devices to be used for data and log volumes. this can be found using the commands "multipath -ll", "lsblk" or "udevadm --query=all --name {device name} | grep | DM_NAME.
- Create or add(depending on the scenario) the following lines to the global.ini file(The below assumes a 2 + 1 configuration, with 2 worker nodes and one standby node):
[persistence] basepath_datavolumes={data volume path - typically /hana/data/<SID>} basepath_logvolumes={log volume path - typically /hana/log/<SID>} use_mountpoints=yes basepath_shared=yes [storage] ha_provider=hdb_ha.fcClient partition_*_*__prtype=5 partition_1_data__wwid={wwid of data volume} partition_1_log__wwid={wwid of log volume} partition_2_data__wwid={wwid of data volume} partition_2_log__wwid={wwid of log volume}Note the use of "partition_*_*__prtype=5" this informs the storage API connector that the reservation type to use for each volume is Write Exclusive.
- Ensure that all of the volumes to be used for log and data can be seen but not mounted by each host in the SAP HANA scale out landscape.
- To perform the installation from the command line, as the root user pass in the location of the global.ini file (this is assuming a new system installation):
./hdblcm --storage_cfg={path to global.ini}
In SPS05 there is a bug for scale out installation where the data and log path values are not being assigned correctly from the global.ini file.
To workaround this issue use the --datapath= and --logpath= values on the installer command line :
./hdblcm --storage_cfg={path to global.ini} --datapath=<Path to data location> --logpath=<Path to log location>For any more information on the SAP HANA Storage API connector, please see SAP Note 1922823.
File Services - NFS
With a shared file system configuration with NFS, a method to fence the failed node is imperative to prevent multiple hosts accessing the persistence volumes and potentially corrupting data. Not all versions of NFS provide a proper fencing mechanism. Starting with version 4, a lease-time based locking mechanism is available, which can be used for I/O fencing. However, NFS version 3 and older versions do not support locking as required for high availability and therefore, achieve fencing capabilities using the STONITH (“shoot the other node in the head”) method. Even in NFS version 4 environments, STONITH is commonly used to ensure that locks are always freed and to potentially speed up the failover process. In our test environment we used STONITH with NFS versions 3 and 4.
The Storage Connector API was used for invoking the STONITH calls. During failover, the SAP HANA master host calls the STONITH method of the custom Storage Connector with the hostname of the failed host as the input value. SAP HANA’s behavior with an active custom Storage Connector is as follows: