Commit a7b7aa85 authored by Yong Tang's avatar Yong Tang Committed by Derek Murray
Browse files

Add KinesisDataset support for tensorflow Dataset (#19712)



* Add KinesisDataset support for tensorflow Dataset

This fix is an attempt to add Kinesis support
for tensorflow's Dataset. Kinesis is provided by
AWS as a managed data streaming service. It is
similiar to Apache Kafka, often used in places
where maintaining a independent Kafka cluster on AWS
is not desirable or possible.

This fix adds the Kinesis support for tensorflow Dataset.
Similiar to the Kafka integration in tensorflow,
KinesisDataset outputs tf.string for records.

Test cases have also been added, which could be invoked manually.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Expose KinesisDataset in dataset_ops.cc

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Expose KinesisDataset in python wrapper

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Add test cases for KinesisDataset

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Update AWS library include files

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Add Bazel BUILD files

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Rename s3_crypto to aws_crypto

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Rename with_s3_support to with_aws_support

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Selectively add kinesis to tensorflow/contrib/BUILD

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Set different partition key and pylint fix.

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Add missing modules in cmake's python_modules.txt

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>

* Address review feedback

Signed-off-by: default avatarYong Tang <yong.tang.github@outlook.com>
parent 7e8927e7
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment