en-es
en-fr
en-sl
en
0.25
0.5
0.75
1.25
1.5
1.75
2
Efficient I/O and Data Intensive Computing
Published on Sep 19, 20161292 Views
Related categories
Chapter list
Efficient I/O and Data Intensive Computing00:00
Overview01:27
Before Running Applications Involving Large Datasets 02:03
I/O in Compute- or Data-Intensive Applications02:56
Overview - 105:11
Protocols for Data Transfer05:16
Data Transfer Using scp or WinSCP05:53
More Information on Using WinSCP07:18
Data Transfer Using rsync07:20
Using Globus Connect08:00
Data Transfer Issues - Real World Scenario09:23
Overview - 210:07
Culling the Data Collection 10:28
Interactive Visualization for Data Culling and Quality Control10:33
Overview - 311:01
Modern HPC Cluster - 111:08
Modern HPC Cluster - 211:31
Lustre Filesystem at TACC12:33
Lustre File System - Striping13:20
…Parallel Filesystem (e.g., Lustre) Provides the Magic15:38
Lustre File System - Striping on TACC Resources16:28
Lustre Commands16:30
Need for High-Level Support for Parallel I/O18:04
You Can Stress Out Lustre Easily if You…18:14
Overview - 418:22
Typical Pattern: Parallel Programs Doing Sequential I/O18:26
Another Pattern: Each Process Writing to a Separate File19:20
Desired Pattern: Parallel Programs Doing Parallel I/O19:51
MPI for Parallel I/O20:41
Using MPI-I/O20:45
MPI-I/O API Opening and Closing a File20:50
MPI-I/O API for Reading Files20:51
Reading a File: readFile2.c - 120:52
Reading a File: readFile2.c - 220:56
Reading a File: readFile2.c - 320:58
MPI-I/O API for Writing Files21:07
Writing a File: writeFile1.c - 121:08
Writing a File: writeFile1.c - 221:09
Compile & Run the Program on Compute Node21:10
File Views for Writing to a Shared File - 121:11
File Views for Writing to a Shared File - 221:12
Writing a File: writeFile2.c - 121:20
Writing a File: writeFile2.c - 221:21
Compile & Run the Program on Compute Node21:22
Collective I/O - 121:22
Collective I/O - 221:24
MPI-I/O Hints21:25
Lustre - setting stripe count in MPI Code21:25
HDF5 and Parallel HDF521:26
Hierarchical Data Format (HDF)21:33
General Structure of HDF5 Code22:03
HDF5 Code Example - 122:04
HDF5 Code Example - 222:04
Compiling and Running the HDF5 Code22:05
Dump of the Output File from the HDF5 Example Program22:06
Highly Recommend to Explore Parallel HDF523:35
Overview - 523:36
Storage, Archival, and Information Visualization23:37
Overview - 623:41
Hadoop23:57
Wrangler: An XSEDE Resource for Data Intensive Computing 24:24
DSSD Storage24:29
A brief intro to Hadoop and Spark24:31
What is apache Hadoop?24:59
What is Apache Spark?26:13
Hadoop - python scripts for Word Count27:44
Spark-python script for WordCount30:01
References31:58