Transcript PPT - S3Lab
An Efficient and Transparent Transaction Management based on the Data Workflow of HVEM DataGrid Im Young Jung Seoul National University Introduction Transaction Management for a safe data update and insertion on e-Science DataGrid Heterogeneous storages according to the characteristics and the size of data Based on workflow, the storing precedence of data across heterogeneous storages in a transaction In this paper 2 An efficient and transparent transaction management on HVEM DataGrid Dividing the transaction into sub-transactions according to the transaction states and Classifying them Transaction hierarchy and parallelism provide efficient and safe large data upload to HVEM DataGrid transparency in the transaction including simultaneous access to heterogeneous storages Automatic garbage collection HVEM Grid High Voltage Electron Microscope(HVEM) Let scientists realize the 3D structure analysis of new materials in micrometer- scale HVEM Grid 3 Remote users can perform the same tasks as on-site scientists. Remote controlling of HVEM Storing, retrieval and search data through HVEM DataGrid Processing data through HVEM Computational Grid HVEM DataGrid Designed for Biologic experiments using HVEM A logical view of one storage for DB and file storage The small metadata is stored at DB Information for materials, material handling methods, HVEM experiments, Images, 4 experimenters The large files are stored in file storages 2D or 3D image files, the documents related to HVEM experiments Internal process to find files After finding their logical path in the file storage by searching the DB, users can retrieve the files they want in the file storage HVEM DataGrid A unified data management The storing precedence among data When store all biological information for the images, we should keep the images in HVEM Grid at the same time The relational semantics between various data stored in distributed heterogeneous storages To upload many large files to HVEM DataGrid efficiently and safely Upload dependency & Serialization Ensure the transactions for safe 5 parallel uploads An efficient and transparent transaction management Requirement for the transactions on HVEM DataGrid Consider the semantic of HVEM DataGrid A project is composed of several experiments The data for an experiment should be inserted according to its data workflow The file and its metadata should be stored to HVEM DataGrid simultaneously. Otherwise, all of them should be deleted Support the long lifetime transaction according to the timelimit of experiment or project the short lifetime transaction which stores the data to HVEM DataGrid physically The optimization for the upload of large files to reduce the blocking time should ensure safe transactions An asynchronous and parallel upload scheme should protect upload dependency and ensure safe transactions 6 An efficient and transparent transaction management Transaction hierarchy For Project The transaction units as checkpoints on For Experiment incomplete data insertion Confine the rollback extent For a group of TnSs Parallel Processing For storing data to physical storage When the data for an experiment or a project is not inserted to HVEM DataGrid until each timelimit, the experiment or the project should be vanished by the rollback of TnE or TnP TnS((((1)2)5)2) (1) represents the identity of TnP it belongs to The next index ‘2’ indicates the identity of TnE and so on Support Autonomous garbage collection It is dependent on users to insert data or delete it on HVEM DataGrid. When they do not insert experimental data any more due to any reason without deleting the 7 related data, HVEM DataGrid would have a big garbage. Transaction management Scheme HVEM DataGrid forks two processes to connect DB and file storage each. In the light failure(LF) due to temporary failures on network or server, When the connections succeed, it gets the next requests and so on. retry the transaction fixed times When jSiS the jSiD(the notification from DB), jSiF(the notification from the file storage) jSiE (both of retries fail, a serious failure(SF) is assumed rollback process 8 them arrive) : TnS completes The state change of TnS(((())j)i) Evaluation Analysis Transparency Through transaction hierarchy and fine grained state management the transaction manager in HVEM DataGrid enables the transparent transaction to upload the image files to the file storage and store their metadata to DB simultaneously. Serializability Many TnSs are upload serializable because their state changes are logged through transaction index. To keep the upload dependency, the transaction manager protects the first user entering TnW. o If he withdraws the TnW, then an other user can initiate the TnW Transaction performance Support the transaction scheme asynchronism and parallelism Experiment Setting Because the sub-transaction time on DB is negligible compared with that on file storage due to data size, we only considered the upload time for image file Considering the semantic of the data workflow in HVEM DataGrid For an asynchronous file transfer, the request intervals for file transfer are chosen randomly within 50 sec The physical locations of the file storages are assumed to be distributed 9 Evaluation Overhead 10 Log management cost The cost for TnP, TnE and TnW; The general transaction management requires the log for TnS The log size for TnP, TnE and TnW is smaller than that for TnS because they function as checkpoint rather than real transaction units. Rollback cost The cascade rollback of TnS in TnW due to the upload dependency on parallel processing of TnS At LF, if the retry succeeds, the gain from transaction parallelism can be very large especially for large file handling There are not many SFs or LFs because e-Science DataGrid is not popular as the multimedia storage Conclusion A transaction management on HVEM Grid Safety Ensure a safe transaction considering the data workflow in HVEM DataGrid Efficiency Improve the performance to upload large files by asynchronism and parallelism Transparency Data management across the heterogeneous storages Automatic garbage collection Reduce garbage 11