Commit 3e8c4fd7 authored by Vijay Vasudevan's avatar Vijay Vasudevan Committed by TensorFlower Gardener
Browse files

Don't establish contexts on gpus not on visible_device_list.

Moves all the initialization code out of gpu_init.h and into gpu_device.cc,
because we want the code that establishes peer mappings between GPUs to
reside close to where the device selection order is made.

Now the initialization code does nothing but calls the StreamExecutor
platform initialization.

This also checks that there are no duplicate entries in the visible_device_list.

I tested this by running the following program:

import time
import tensorflow as tf

c = tf.ConfigProto()
c.gpu_options.visible_device_list="1"

s = tf.Session(config=c)
time.sleep(5)

# nvidia-smi showed the context was established on device 1 but NOT 0

del s
c.gpu_options.visible_device_list="1,0"
s = tf.Session(config=c)
time.sleep(30)

# nvidia-smi showed the context was established on both device 0 and 1,
# and the logs showed that the device ordering was 1->/gpu:0 and 0->/gpu:1,
# as well as the fact that it tried to establish the peer mapping.

del s
c.gpu_options.visible_device_list="1,0,1"
s = tf.Session(config=c)  # failed

Fixes #1888
Change: 131785661
parent 9d646782
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment