Content OverviewSetupOverviewLogging device placementManual device placementLimiting GPU memory growthUsing a single GPU on a multi-GPU systemUsing multiple GPU\TensorFlow code, and tf.keras models will transparently run on a single GPU with no code changes required.:::tipNote: Use tf.config.list_physical_devices('GPU') to confirm that TensorFlow is using the GPU.:::The simplest way to run on multiple GPUs, on one or many machines, is using Distribution Strategies.This guide is for users who have tried these approaches and found that they need fine-grained control of how TensorFlow uses the GPU. To learn how to debug performance issues for single and multi-GPU scenarios, see the Optimize TensorFlow GPU Performance guide.SetupEnsure you have the latest TensorFlow gpu release installed.\import tensorflow as tfprint("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))\2024-08-15 02:53:40.344028: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered2024-08-15 02:53:40.365851: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered2024-08-15 02:53:40.372242: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registeredNum GPUs Available: 4WARNING: All log messages before absl::InitializeLog() is called are written to STDERRI0000 00:00:1723690422.944962 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.948934 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.952655 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.955880 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.967120 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.970596 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.973980 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.976984 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.979869 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.983344 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.986754 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690422.989690 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355OverviewTensorFlow supports running computations on a variety of types of devices, including CPU and GPU. They are represented with string identifiers for example:"/device:CPU:0": The CPU of your machine."/GPU:0": Short-hand notation for the first GPU of your machine that is visible to TensorFlow."/job:localhost/replica:0/task:0/device:GPU:1": Fully qualified name of the second GPU of your machine that is visible to TensorFlow.If a TensorFlow operation has both CPU and GPU implementations, by default, the GPU device is prioritized when the operation is assigned. For example, tf.matmul has both CPU and GPU kernels and on a system with devices CPU:0 and GPU:0, the GPU:0 device is selected to run tf.matmul unless you explicitly request to run it on another device.If a TensorFlow operation has no corresponding GPU implementation, then the operation falls back to the CPU device. For example, since tf.cast only has a CPU kernel, on a system with devices CPU:0 and GPU:0, the CPU:0 device is selected to run tf.cast, even if requested to run on the GPU:0 device.Logging device placementTo find out which devices your operations and tensors are assigned to, put tf.debugging.set_log_device_placement(True) as the first statement of your program. Enabling device placement logging causes any Tensor allocations or operations to be printed.\tf.debugging.set_log_device_placement(True)# Create some tensorsa = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])c = tf.matmul(a, b)print(c)\Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0I0000 00:00:1723690424.215487 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.217630 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.219585 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.221664 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.223723 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.225666 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.227528 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.229544 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.231494 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.233433 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.235295 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.237325 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.276919 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.278939 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.280845 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.282884 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.284977 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.286923 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.288779 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.290783 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.292741 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.295170 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.297460 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355I0000 00:00:1723690424.299854 162671 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0tf.Tensor([[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32)The above code will print an indication the MatMul op was executed on GPU:0.Manual device placementIf you would like a particular operation to run on a device of your choice instead of what's automatically selected for you, you can use with tf.device to create a device context, and all the operations within that context will run on the same designated device.\tf.debugging.set_log_device_placement(True)# Place tensors on the CPUwith tf.device('/CPU:0'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])# Run on the GPUc = tf.matmul(a, b)print(c)\Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0tf.Tensor([[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32)You will see that now a and b are assigned to CPU:0. Since a device was not explicitly specified for the MatMul operation, the TensorFlow runtime will choose one based on the operation and available devices (GPU:0 in this example) and automatically copy tensors between devices if required.Limiting GPU memory growthBy default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation. To limit TensorFlow to a specific set of GPUs, use the tf.config.set_visible_devices method.\gpus = tf.config.list_physical_devices('GPU')if gpus: # Restrict TensorFlow to only use the first GPU try: tf.config.set_visible_devices(gpus[0], 'GPU') logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU") except RuntimeError as e: # Visible devices must be set before GPUs have been initialized print(e)\Visible devices cannot be modified after being initializedIn some cases it is desirable for the process to only allocate a subset of the available memory, or to only grow the memory usage as is needed by the process. TensorFlow provides two methods to control this.The first option is to turn on memory growth by calling tf.config.experimental.set_memory_growth, which attempts to allocate only as much GPU memory as needed for the runtime allocations: it starts out allocating very little memory, and as the program gets run and more GPU memory is needed, the GPU memory region is extended for the TensorFlow process. Memory is not released since it can lead to memory fragmentation. To turn on memory growth for a specific GPU, use the following code prior to allocating any tensors or executing any ops.\gpus = tf.config.list_physical_devices('GPU')if gpus: try: # Currently, memory growth needs to be the same across GPUs for gpu in gpus: tf.config.experimental.set_memory_growth(gpu, True) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Memory growth must be set before GPUs have been initialized print(e)\Physical devices cannot be modified after being initializedAnother way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.The second method is to configure a virtual GPU device with tf.config.set_logical_device_configuration and set a hard limit on the total memory to allocate on the GPU.\gpus = tf.config.list_physical_devices('GPU')if gpus: # Restrict TensorFlow to only allocate 1GB of memory on the first GPU try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e)\Virtual devices cannot be modified after being initializedThis is useful if you want to truly bound the amount of GPU memory available to the TensorFlow process. This is common practice for local development when the GPU is shared with other applications such as a workstation GUI.Using a single GPU on a multi-GPU systemIf you have more than one GPU in your system, the GPU with the lowest ID will be selected by default. If you would like to run on a different GPU, you will need to specify the preference explicitly:\tf.debugging.set_log_device_placement(True)try: # Specify an invalid GPU device with tf.device('/device:GPU:2'): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c = tf.matmul(a, b)except RuntimeError as e: print(e)\Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2If the device you have specified does not exist, you will get a RuntimeError: .../device:GPU:2 unknown device.If you would like TensorFlow to automatically choose an existing and supported device to run the operations in case the specified one doesn't exist, you can call tf.config.set_soft_device_placement(True).\tf.config.set_soft_device_placement(True)tf.debugging.set_log_device_placement(True)# Creates some tensorsa = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])c = tf.matmul(a, b)print(c)\Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0tf.Tensor([[22. 28.] [49. 64.]], shape=(2, 2), dtype=float32)Using multiple GPUsDeveloping for multiple GPUs will allow a model to scale with the additional resources. If developing on a system with a single GPU, you can simulate multiple GPUs with virtual devices. This enables easy testing of multi-GPU setups without requiring additional resources.\gpus = tf.config.list_physical_devices('GPU')if gpus: # Create 2 virtual GPUs with 1GB memory each try: tf.config.set_logical_device_configuration( gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024), tf.config.LogicalDeviceConfiguration(memory_limit=1024)]) logical_gpus = tf.config.list_logical_devices('GPU') print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs") except RuntimeError as e: # Virtual devices must be set before GPUs have been initialized print(e)\Virtual devices cannot be modified after being initializedOnce there are multiple logical GPUs available to the runtime, you can utilize the multiple GPUs with tf.distribute.Strategy or with manual placement.With tf.distribute.StrategyThe best practice for using multiple GPUs is to use tf.distribute.Strategy. Here is a simple example:\tf.debugging.set_log_device_placement(True)gpus = tf.config.list_logical_devices('GPU')strategy = tf.distribute.MirroredStrategy(gpus)with strategy.scope(): inputs = tf.keras.layers.Input(shape=(1,)) predictions = tf.keras.layers.Dense(1)(inputs) model = tf.keras.models.Model(inputs=inputs, outputs=predictions) model.compile(loss='mse', optimizer=tf.keras.optimizers.SGD(learning_rate=0.2))\INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3')Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op FloorMod in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Cast in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op StatelessRandomGetKeyCounter in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:2output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:2product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:1output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:3output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0FloorMod: (FloorMod): /job:localhost/replica:0/task:0/device:GPU:0z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0Cast: (Cast): /job:localhost/replica:0/task:0/device:GPU:0y_RetVal: (_DeviceRetval): /job:localhost/replica:0/task:0/device:GPU:0input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0seed: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0StatelessRandomGetKeyCounter: (StatelessRandomGetKeyCounter): /job:localhost/replica:0/task:0/device:GPU:0key_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0counter_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0key: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0counter: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0alg: (_DeviceArg): /job:localhost/replica:0/taExecuting op StatelessRandomUniformV2 in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0Executing op AddV2 in device /job:localhost/replica:0/task:0/device:GPU:0Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1sk:0/device:CPU:0StatelessRandomUniformV2: (StatelessRandomUniformV2): /job:localhost/replica:0/task:0/device:GPU:0output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0Sub: (Sub): /job:localhost/replica:0/task:0/device:GPU:0z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0Mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0x: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0y: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0AddV2: (AddV2): /job:localhost/replica:0/task:0/device:GPU:0z_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3NoOp: (NoOp): /job:localhost/replica:0/task:0/device:GPU:0dims: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0Fill: (Fill): /job:localhost/replica:0/task:0/device:GPU:0output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1VarHandleOp: (VarHandleOp): /job:lExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0ocalhost/replica:0/task:0/device:GPU:1resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:0resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:1output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:1AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:1input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:2output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:2AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:2input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3value: (_Arg): /job:localhost/replica:0/task:0/device:GPU:3AssignVariableOp: (AssignVariableOp): /job:localhost/replica:0/task:0/device:GPU:3resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:0resource: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0ReadVariableOp: (ReadVariableOp): /job:localhost/replica:0/task:0/device:GPU:0value_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:1VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:1resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:2VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:2resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/device:GPU:3resource_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0VarHandleOp: (VarHandleOp): /job:localhost/replica:0/task:0/deviExecuting op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:1Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:2Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:2Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op Identity in device /job:localhost/replica:0/task:0/device:GPU:3Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:3Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0Executing op NoOp in device /job:localhost/replica:0/task:0/device:GPU:0This program will run a copy of your model on each GPU, splitting the input data between them, also known as "data parallelism".For more information about distribution strategies, check out the guide here.Manual placementtf.distribute.Strategy works under the hood by replicating computation across devices. You can manually implement replication by constructing your model on each GPU. For example:\tf.debugging.set_log_device_placement(True)gpus = tf.config.list_logical_devices('GPU')if gpus: # Replicate your computation on multiple GPUs c = [] for gpu in gpus: with tf.device(gpu.name): a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]) c.append(tf.matmul(a, b)) with tf.device('/CPU:0'): matmul_sum = tf.add_n(c) print(matmul_sum)\Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:1Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:1Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:2Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:2Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:3Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:3Executing op AddN in device /job:localhost/replica:0/task:0/device:CPU:0tf.Tensor([[ 88. 112.] [196. 256.]], shape=(2, 2), dtype=float32)\\:::infoOriginally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License.:::\