Support S3 Filesystem for Tensorflow (#11089)
* Add AWS C++ SDK to bazel BUILD file This commit adds AWS C++ SDK to bazel BUILD file, as part of the effort to support AWS S3 for TensorFlow. Add LICENSE file for AWS C++ SDK Adds srcs to bazel BUILD based on platform for AWS C++ SDK This commit adds srcs to bazel BUILD based on platform for AWS C++ SDK so that only related platform files (like in Linux) are built. Signed-off-by:Yong Tang <yong.tang.github@outlook.com> * Add stub s3 file system for TensorFlow This commit adds stub s3 file system for TensorFlow. Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Add basic S3 file system for TensorFlow This fix adds basic S3 file system for TensorFlow. Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Add WritableFileSystem for TensorFlow S3 file system This fix adds WritableFileSystem for TensorFlow S3 file system. Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Add ReadOnlyMemoryRegion and AppendableFileSystem for S3. This fix adds implementation of ReadOnlyMemoryRegion and AppendableFile systems for S3. Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Add implementation of S3 File System. Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Get S3 configurations from environment variables to support 3rd-party S3 implementations * Use old ListObject (V1) instead of ListObjectV2 to support some 3rd-party S3 implementations * A workaround for the Invalid Range Error when reading ImageNet data set * Sanitize s3_file_system.cc with clang-format. Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Split the header and source of s3_file_system.cc * Fix a return status issue in RandomAccessFile::Read * Add the unit test of S3 file system * Change @protobuf to @protobuf_archive And reformat with clang-format and additional cleanup Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Address review comments Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Fix bazel build error on darwin. Signed-off-by:
Yong Tang <yong.tang.github@outlook.com> * Update sha256sum for aws cpp sdk download. Signed-off-by:
Yong Tang <yong.tang.github@outlook.com>
Loading
Please sign in to comment