An Efficient Recovery Technique using Global Buffer on SAN Environments

CHOON SEO PARK, GYOUNG BAE KIM, YONG JU LEE, HAG-YOUNG KIM, KEE-WOOK RIM
Dept. Of Computer System, Computer & Software Technology Labs
ETRI(Electronics and Telecommunications Research Institute) 
161 Kajong-Dong, Yusong-Gu, Taejon, 305-350 
KOREA
{parkcs,  gbkim,  yongju, h0kim, rim}@etri.re.kr

Abstract: The shared disk file systems use a technique known as file system journaling to support recovery of metadata on the SAN(Storage Area Network). In the existing journaling technique, the metadata that is dirtied by one host must be updated to disk space before some hosts access it. The system performance is decreased because the disk access number is increased. In this paper, we describe a new recovery technique using a global buffer to decrease disk I/O. It transmits the dirtied metadata into the other hosts through Fibre Channel network on the SAN instead of disk I/O and supports recovery of a critical data by journaling a data as well as metadata

Key-Words: SAN (Storage Area Network), recovery, journaling, metadata, global buffer, Fibre Channel network

1  Introduction
Recently, it is necessary to what large volumes of data are stored on file system according to extension of Internet and fast variation of work environments. However, in the existing large single system, it exceeds the bound of manipulating large amount of data that is increased by geometric progression.
A new architecture called the SAN was developed in response to the requirements. The SAN is that large volumes of data can be located independent of any one computer system and yet be available to all servers that have need of it. All computers share distributed storage devices by one network using fibre channel [1]. This SAN environment provides high performance, high scalability, high availability and sharability. There are such as GFS (Global File System)[2-3], Veritas [4], SANergy [5] and so on, on SAN environment.
In this SAN environment, we use a cashed data that are metadata and real data, without writing dirtied data to on the disk to improve system performance. However, it is possible to become inconsistent state when system crashes. A traditional means of recovery has been to run a file system check routine (fsck) that checks and corrects file system inconsistencies. But, the problem with file system check routines is that they are slow because they take time proportional to the size of the file system, the file system must be off-line while the fsck is being performed and, therefore, this technique is unacceptable for shared file systems. Instead, we uses a technique known as file system journaling to avoid fsck’s altogether and reduce recovery time and increase availability [6-9].
The journaling technique is that it writes log of metadata such as super block, directory entry, inode information and so on, to on the journal spaces for recovery data when system crashes. In the event of a system failure, a file system is restored to a consistent state by replaying the log on journal space and applying log records for the appropriate transactions.
GFS file system uses a journaling technique. However, in the traditional journaling technique, the metadata that is dirtied by one host must be updated to disk space before some hosts access it. So, the system performance decreases because the disk access number increases. In this paper, we propose a new recovery technique using a global buffer to decrease disk I/O to solve this disadvantage. It transmits the dirtied metadata into the other hosts through Fibre Channel network on the SAN instead of disk I/O. we improve system performance because transmission by network is faster than disk I/O.  Also, there is disadvantage that doesn’t support journaling real data on traditional journaling techniques. We support recovery of a critical data by journaling a data as well as metadata.

2  Related Works
In this paragraph, we describe characteristics of fsck that is recovery of traditional Linux file systems. Also, characteristics of JFS and GFS file systems to recover disadvantage of fsck.

2.1	FSCK (File System Check Routine)
The role of file system is that stores and searches data. The file system has internal data structure, which is metadata, to all data is organized and accessed. Metadata is structured data that describes the characteristics of a resource.
In traditional Linux file systems use cash for metadata to improve system performance. But, it is possible to become inconsistent state when a single machine crashes. The file systems must be consistent state for metadata. There is a traditional means of recovery has been to run a file system check routine (fsck).
It checks and corrects file system inconsistencies. But, the problem with file system check routines is that they are slow because they take time proportional to the size of the file system, the file system must be off-line while the fsck is being performed and, therefore, this technique is unacceptable for shared file systems. Instead, SANtopia uses a technique known as file system journaling to avoid fsck’s altogether and reduce recovery time and increase availability.

2.2	JFS (Journaled File System)
JFS file systems use a technique known as file system journaling to support recovery of metadata. JFS provides fast file system restart in the event of a system crash. Using file systems journaling techniques, JFS can restore a file system to a consistent state in a matter of seconds or minutes, versus hours or days with non-journaled file systems.
The JFS file systems write log of updated metadata to journal space. JFS file systems keep these operations atomic, so that file system must be maintained consistent state. In the event of a system failure, a file system is restored to a consistent state by replaying the log on journal space and applying log records for the appropriate transactions. The recovery time associated with this log-based approach is much faster since the replay utility need only examine the log records produced by recent file system activity rather than examine all file system meta-data. However, because The JFS file systems is local file systems on single Linux environment, this technique is unacceptable for shared file systems on SAN environment.
2.3	GFS (Global File System)
The GFS file system is shared disk file system on SAN environments. The GFS uses a technique known as journaling to support recovery of metadata. The GFS file system reduce recovery time by journaling technique and the file system is possible on-line while the recovery is being performed. The GFS file systems has it own journal space, that is protected by one lock that is acquired at mount time and released at unmount (or crash) time. Each journal can be on its own disk for greater parallelism. Each journal must be visible to all clients for recovery.
The GFS file systems use transaction and log manager for journaling technique. In the metadata update operation, operation must be atomic by transaction. The log manager write updated log to own journal space. In the GFS file systems, the metadata that is dirtied by one host must be updated to disk space before some hosts access it. The system performance is decreased because the disk access number is two after metadata is updated. That is, The GFS file systems has disadvantage that it write log to journal space and metadata to disk space. For example, in Fig.1, these operations are shown

Fig.1 In the GFS file systems, journaling technique of metadata
A host-A reads metadata from the disk space ①. The host-A modifies the metadata and writes journal space ②. When host-B access metadata that is modified by host-A, host-A writes real metadata to disk space ③ . After modified metadata is written to disk, Host-B reads metadata from disk space ④. Host-B modify metadata and writes log to journal space of host-B ⑤. The disk I/O number is five, so the system performance is decreased.
In the GFS file systems not write metadata log to disk space but asynchronous logging. The asynchronous logging has advantage reduce number of disk I/O. However, when transaction is committed but write log journal spaces, the committed transaction was ignored. Also, there is disadvantage that doesn’t support journaling real data on traditional journaling techniques.

3 The efficient recovery technique using global buffer on SAN environments

In this paragraph we describe configuration of recovery manger proposed by in this paper and explain each manager. When other host access modified metadata, we propose efficient journaling technique using global buffer to overcome disadvantage of GFS file system. Also, we overcome asynchronous I/O and provide recovery of a critical data by journaling a data as well as metadata.

3.1 The structure of recovery manager
In this paragraph, we describe recovery module of metadata on SAN environment. In Fig.2, this structure is shown.

Fig.2 The structure of recovery on SAN environments
The structure of recovery is composed of transaction manager for performing each operation, log manager for logging log and recovery manager for recovering metadata.

3.2 The Transaction Manager
The journaling technique use transactions for operations that change the metadata and real data state. These operations must be atomic, so that file system must be maintained consistent state. We treat unit of recovery operation as transaction that is consecutive operation.  
In Fig.3, progress of transaction is shown. The Transaction manager allocates transaction space and initializes that. Acquire lock for metadata to modify and add it to transaction. That is, transaction manager keep locks and release locks until transaction is end. After pinning metadata buffer that isn’t written to disk, modify metadata. And Log manager is passed a structure that contains a list of modified metadata. The transaction manager releases the locks and other host access the metadata buffer.

Transaction Manager
{
Transaction begin
{
Allocate transaction memory;
Initialize the transaction;
Assign the transaction type;
}
Acquire lock for data;
Add updated data buffer to transaction;
Pinning the data buffer;
Modify metadata buffer;
Transaction end
{
Pass updated metadata to log manager;
Releases the lock;
}
}
}	
Fig.3 Progress of Transaction Manger

In the GFS file systems, the metadata that is modified by one host must be updated to disk space before some hosts access it. So, we get advantage of simple journaling. However, system performance is decreased because of disk I/O. To overcome this disadvantage, in this paper, we propose journaling technique using global buffer.
The global buffer manages all local buffers on all hosts. In Fig.4, progress of journaling is shown.
A host-A reads metadata from the disk space ①. The host-A modifies the metadata and writes journal space    ②. When host-B access metadata that is modified by host-A, host-A asks global buffer manager. Global buffer manager acquires lock for metadata and transmit it to host-B by network channel ③. Host-B modifies metadata and writes log to journal space of host-B ④.
 According to this algorithm, the disk access number is 3, which is smaller than file system of Fig.2. We improve system performance because transmission time by network is faster than disk I/O[10].
 
Fig.4 In the SAN environments, journaling technique of metadata

3.3 The Log Manager
The log manager write log that is delivered by transaction manager to journal space. After the log is logged to journal space, the modified metadata is written to disk space (Write-Ahead Protocol [11]). In the event of system failure, a file system is restored to a consistent state by replaying the log on journal space and applying log records for the appropriate transactions. If the file system shares journal space, recovery technique would be too complex and inefficient. To overcome this disadvantage, in this paper, the file systems has it own journal space, that is protected by one lock that is acquired at mount time and released at unmount (or crash) time. Each journal can be on its own disk for greater parallelism. Each journal must be visible to all clients for recovery. A file system is returned to a consistent state by replaying the log on journal of systems failure quickly.
In the GFS file system uses asynchronous logging for metadata, it can improve system performance, however, when transaction is committed but write log journal spaces, if a system crashes, the committed transaction was ignored. To this disadvantage, in this paper, we proposed that the logging technique of metadata is synchronous logging.
Also, there is disadvantage that doesn’t support journaling real data on traditional journaling techniques such as GFS file systems. In this paper, we support recovery of a critical data by journaling a data as well as metadata. The data buffer has version number like metadata for recovery of real data. The logging technique of real data is asynchronous logging, there is advantage that reduce disk I/O. There is disadvantage that a synchronous logging sync log to disk by transaction commits. So, if we perform synchronous logging for real data, system performance is decrease. Therefore, we perform asynchronous logging for real data. We get good performance because of low disk I/O by using asynchronous logging. Therefore, we support recovery of a critical data by journaling a data as well as metadata.
In Fig.5, progress of Log Manager is shown. Get modified metadata from Transaction Manager and reserve log space. Increase version number of metadata buffer or data buffer. The version number is used by Recovery Manager. If modified data is metadata, perform synchronous logging. Otherwise, perform asynchronous logging. Put information of transaction in AIL(Active Item List) and unpin metadata or data buffer. Later, when the metadata is on disk, remove it from the AIL. The role of AIL is that manage the buffers on the memory after transaction commits.

Log Manager
{
Get the transaction from the Transaction Manager;
Reserve the log space;
Increase version number;
If (modified log == metadata)
{
Synchronous logging;
}
else
{
Asynchronous logging;
}
Put information of transaction in the AIL
(Active Item List);
Unpin the metadata;
Later, when the metadata is on disk, remove it from the AIL;
}
}	
Fig.5  Progress of Log Manger

3.4 The Recovery Manager
In the event of system failure, Recovery Manager is called for consistent state of metadata. The Recovery Manager performs recovery of metadata by using log information on journal space.
Recovery Manager is called with the failed host’s ID. In Fig.6, progress of Recovery Manager is shown. At first, the Recovery Manager reads log head and check what the systems was shutdown uncleanly. If the system was shutdown cleanly, the Recovery Manager is end. Otherwise, search the tail and head entries of journal and ignore partially-commit entries. Acquire log for journal entry associated recovery and compare version number of log on journal space with version number of real data on disk space. If the version number of log on journal space is larger than that of real data on disk, write log to disk space. That is, we perform recovery of data by using replay journal. Next, clear journal space and Recovery Manager is end. Recovery Manager can continue to wrok during recovery unless they need a lock held by a failed host.

Recovery Manager
{
Read log head;
If (were the systems shutdown uncleanly?)
Recovery Manager end;
else
{
The tail and head entries of the journal are found;
Partially-committed entries are ignored;
for(start = first journal entry; 
start != end of journal entry; start = next journal entry)
{
Acquire lock for journal entry;
Read metadata from disk;
Read log from journal space;
If (version umber of metadata on disk <
version number of log on journal space)
{
Replay log to disk;
}
else
{
skip；
}
}
}
  Clear journal space;
}	
Fig.6 Progress of Recovery Manger
3.5 A Scenario of Recovery on SAN environments 
In Fig.7, a scenario of Recovery Manager is shown.
We describe briefly environments of example.

(1)	As in Fig.7, a host is connected with the other host and there are disk space for real data and journal space for log of metadata.
(2)	Each host has own journal space for log and each journal must be visible to all hosts for recovery.

A scenario of recovery proceeds as follows: Host-A reads metadata from disk. We assume version number of metadata is zero ①.
We treat transaction for modifying the metadata buffer and increase number of version; zero became to one. We write log of metadata to journal space of Host-A②. When Host-B access metadata of host-A, Host-A release lock for metadata and transmit it to Host-B. The metadata is transmitted to Host-B by Host-A, version number of metadata increases ③.
In this example, when Host-B write log to journal space and Host-A and Host-B don’t write data to disk, Host-A crashes④

Fig.7 Scenario of Recovery Manger
The recovery is called with the failed host’s ID. First, it reads log of journal of Host-A and data of disk. It compare log with data of disk. Because version of log is 1 and that of data is 0, it replay log. Therefore, version of disk became 1 ⑤. 

4 Implementations Journaling of Shared Disk File System on SAN environments
In this paragraph, we describe implementations journaling of shared disk file systems on SAN environments. The recovery technique is implemented as SANtopia of ETRI. We describe environment of SANtopia and implementations of Transaction Manager, Log Manager and Recovery Manager on SANtopia.

4.1 The Implementations environments of SANtopia Recovery Manager
In table 1, the implementations environments of SANtopia file systems is shown.

Table 1 Implementations environments of SANtopia
Linux kernel version：2.2.18
The default size of block：4096 byte - 4 k byte
The default size of journal：128 MB
The default size of journal segment：16 blocks
The default number of journal：1	

The kernel version is 2.2.18 and the default size of block 4kb byte. The default size of journal is 128 MB and there are journal the same number of host on SAN environment. The default size of journal segment is 16 blocks.

4.2 The Implementations of Transaction Manager
The Transaction Manager manages AIL for metadata and lock information.
In Fig.8, process of symbolic link, which is system call operation, is shown. This operation is treated by transaction because it takes place variation of metadata. We observe carefully this operation, at first, check authority and search directory entry. A sfs_trans_begin function is called and allocates transaction. Acquire lock for accessing metadata and add it to transaction. Transaction Manager manages the locks. Next, change link count and a sfs_trans_end is called. It deliver log to Log Manager and release the lock.

Fig.8 Process of symbolic link of SANtopia file systems  

4.3 The Implementations of Log Manager
The host on SANtopia file systems has own journal space and each host manages journal space. In Fig.9, the structure of journal is shown. The structure is composed of log header, log descriptor and metadata. 
In Fig.9, the journaling technique is shown. It read metadata from disk space and add it to transaction.
When the operation for variation of metadata, Log Manager allocate control buffer for logging of location of real metadata. Also, we allocate metadata log buffer for writing of information of real metadata. The log buffer is composed of control buffer and data buffer
The control buffer is log header and location of real metadata on disk space. The data buffer is log of real metadata for logging it to journal space.

Fig.9 The structure of journal and process of journaling of SANtopia file systems

4.4 The Implementations of Recovery Manager
When system crashes, sfs_recovery function is called. If the version number of log on journal space is larger than that of real data on disk, write log to disk space. That is, we perform recovery of data by using replay journal.

5    Conclusions and Future Work
In this paper, we described journaling for recovery of data when system crashes on SAN environments. In the traditional GFS file systems, the metadata that is dirtied by one host must be updated to disk space before some hosts access it. The system performance is decreased because the disk access number is increased. To this overcome disadvantage, we propose a new recovery technique using a global buffer to decrease disk I/O. It transmits the dirtied metadata into the other hosts through Fibre Channel network on the SAN instead of disk I/O.
In the GFS file system, asynchronous logging can improve system performance, however, when transaction is committed but a log is written to journal spaces, the committed transaction was ignored. To this disadvantage, in this paper, we proposed that the logging technique of metadata is synchronous logging. Also, we support recovery of a critical data by journaling a data as well as metadata.
If we perform synchronous logging for real data, system performance is decrease. Therefore, we perform asynchronous logging for real data. We get good performance because of low disk I/O by using asynchronous logging.
In this paper, there is problem that size of log is increase according to size of file increase because the proposed journaling technique is that it write buffer itself. In the future, we should search journaling technique of small size log.

References:
[1] Clit Jurgens, Fibre Channel：A Connection to the Future, IEEE Computer, Vol.28, No.8, August, 1995, pp.82-90
[2] Kenneth W. Preslan, et al., A 64-bit, Shared Disk File System for Linux, Proceedings of the 16th IEEE Mass Storage Systems Symposium, San Diego, California, March, 1999, pp.22-41
[3] Kenneth W. Preslan, et al., Implementing Journaling in a Linux Shared Disk File Systems, Proceedings of the 17th IEEE Mass Storage Systems Symposium, College Park, Maryland, March, 2000, pp.351-378
[4] http：//www.veritas.com.
[5] http：//www.tivoli.com/products/index/
sanergy_file_sharing/.
[6] http：//www-124.ibm.com/developerworks/
oss/jfs/.
[7] http：//oss.sgi.com/projects/xfs/.
[8] Chandramohan A. Thekkath, Timothy Mann, and Edward K. Lee, Frangipani：A Scalable Distributed File System, Proceedings of the 16th ACM Symposium on Operating Systems Principles, St. Malo, France, October, 1997, pp.224-237
[9] Stephen C. Tweedie, Journaling the Linux ext2fs Filesystem, Proceeding of the LinuxExpo'98.
[10] C, Mohan Inderpal Narang, Recovery and Coherency- Control Protocols for Fast Intersystem Page Transfer and Fine-Granularity Locking in a Shared Disks Transaction Environment, Proceedings of the 17th International Conference on Very Large Data Bases, Barcelona, September, 1991, pp.193-207
[11] Philip A. Bernstein and Eric Newcomer, Principles of Transaction Processing, morgan Kaufmann Publishers, 1997.