Commit f6a061bb authored by Yong Tang's avatar Yong Tang Committed by gunan
Browse files

Support S3 Filesystem for Tensorflow (#11089)



* Add AWS C++ SDK to bazel BUILD file

This commit adds AWS C++ SDK to bazel BUILD file,
as part of the effort to support AWS S3 for TensorFlow.

Add LICENSE file for AWS C++ SDK

Adds srcs to bazel BUILD based on platform for AWS C++ SDK

This commit adds srcs to bazel BUILD based on platform
for AWS C++ SDK so that only related platform files (like in Linux)
are built.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Add stub s3 file system for TensorFlow

This commit adds stub s3 file system for TensorFlow.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Add basic S3 file system for TensorFlow

This fix adds basic S3 file system for TensorFlow.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Add WritableFileSystem for TensorFlow S3 file system

This fix adds WritableFileSystem for TensorFlow S3 file system.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Add ReadOnlyMemoryRegion and AppendableFileSystem for S3.

This fix adds implementation of ReadOnlyMemoryRegion and AppendableFile
systems for S3.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Add implementation of S3 File System.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Get S3 configurations from environment variables to support 3rd-party S3 implementations

* Use old ListObject (V1) instead of ListObjectV2 to support some 3rd-party S3 implementations

* A workaround for the Invalid Range Error when reading ImageNet data set

* Sanitize s3_file_system.cc with clang-format.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Split the header and source of s3_file_system.cc

* Fix a return status issue in RandomAccessFile::Read

* Add the unit test of S3 file system

* Change @protobuf to @protobuf_archive

And reformat with clang-format and additional cleanup

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Address review comments

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Fix bazel build error on darwin.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Update sha256sum for aws cpp sdk download.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>
parent b8614b50
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment