TensorFlow🔗
TensorFlow is a popular machine learning (ML) framework.
A common use case is to import TensorFlow as a module in Python.
It is then up to you as a user to write your particular ML application as a Python script using the tensorflow Python module functionality.
We provide precompiled optimised installations of TensorFlow in our tree of software modules.
Just like with most software, search for all available versions with module spider tensorflow.
If you want to run on CUDA accelerated GPU hardware, make sure to select a version with CUDA.
It is also possible to run TensorFlow using containers of which we provide many versions already centrally installed.
TensorFlow is heavily optimised for GPU hardware so we recommend using the CUDA version and to run it on the compute nodes equipped with GPUs. How to do this is described in our guide to running jobs.
After loading a TensorFlow module your environment is now configured to start calling TensorFlow from Python.
Here is a small test that prints the TensorFlow version available in your environment:
[cid@vera ~]$ python -c "import tensorflow as tf; print(tf.__version__)"
2.7.1
If you intend to run your calculations on GPU hardware it can be useful to check that TensorFlow detects the GPU hardware using its device_lib submodule.
Here is an example from a node equipped with a Nvidia Quadro GPU.
[cid@vera ~]$ python -c "from tensorflow.python.client import device_lib; device_lib.list_local_devices()"
...
2020-06-01 16:08:00.418439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:3b:00.0 name: Quadro P2000 computeCapability: 6.1
coreClock: 1.4805GHz coreCount: 8 deviceMemorySize: 4.94GiB deviceMemoryBandwidth: 130.53GiB/s
...
2020-06-01 16:08:00.434406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-06-01 16:08:00.434436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-06-01 16:08:00.926384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-01 16:08:00.926435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-06-01 16:08:00.926442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-06-01 16:08:00.927639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/device:GPU:0 with 4454 MB memory) -> physical GPU (device: 0, name: Quadro P2000, pci bus id: 0000:3b:00.0, compute capability: 6.1)
Dataloading🔗
You find our general dataloading tips at datasets.
Debugging🔗
TensorFlow is a complex software and the stochastic nature of many machine
learning algorithms together with the graph-based functioning of TensorFlow
makes debugging TensorFlow code more challenging than traditional imperative
code. For example, if you have previous debugging experience you are likely
aware that inserting print-statements at a few well-considered points in your
code can be a very efficient way to debug, however in TensorFlow this common
technique will not work in the general case due to the construction and
execution of graphs, which may ignore your print, or only execute it once
during graph initialisation. Even the common approach of attaching an
interactive (or step) debugger, such as gdb or pdb, have also limited
effectiveness in debugging TensorFlow code, due to the aforementioned reasons.
Of course, all hope is not lost - we just have to adjust our tools and
techniques.
This guide provides recommendations to ease the process of debugging TensorFlow 2 using Python and the bundled tools included in the TensorFlow installation available at C3SE.
- Eager vs. Graph execution
- Printing with
tf.print - Increase log severity
- Checkpoint your code
- Visualise using TensorBoard
- Check for
NaNand+-Infinity - Experimental API debugging
The examples shown here have not been tested in Jupyter notebooks. The code
should run but some output may not be visible in the Jupyter notebook. For
example, tf.print outputs to stderr, rather than the notebook. Keeps this in
mind if you are running Jupyter.
Debugging in TensorFlow, experimental APIs, and references to documentation
Debugging is about analysing code that does not behave in an
expected way. Debugging in TensorFlow could be to find numerical issues, for
example NaN or +- Infinity, finding mathematical modelling errors, as well
as trying to answer more broad questions Why is my prediction wrong? or Why
do the predictions differ between these two models?. This guide is limited
in-scope towards the lower-level details of your TensorFlow code, but debugging
covers a wide range of issues, and the tools and API for one type of debugging
might not fit the next.
TensorFlow is still under intense development and breaking public API changes
may occur between major versions. Debugging and profiling features both enter
and transition from the tf.experimental namespace. Always note what version
of TensorFlow you are running, and check what new features you get, and what
changes, by upgrading to the latest version.
Are you running on the GPU?
Are you reading this guide because you are unsure if you are using the GPUs? You can add the following line to the top of your program to find out what devices get mapped to your operations.
import tensorflow as tf
tf.debugging.set_log_device_placement(True)
def square(x):
return tf.math.sqrt(x)
roots = square(tf.constant([2.0,4.0,8.0]))
If we run this code:
$ python example.py
[...]
2021-03-09 09:28:44.485472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 13592 MB memory) -> physical GPU (device: 3, name: Tesla T4, pci bus id: 0000:af:00.0, compute capability: 7.5)
2021-03-09 09:28:44.486117: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2021-03-09 09:28:44.606224: I tensorflow/core/common_runtime/eager/execute.cc:611] Executing op Sqrt in device /job:localhost/replica:0/task:0/device:GPU:3
You will only get one print per eager execution (see further down this article for an explanation of eager vs. graph execution) you will thus not flood your logfiles if you do repeat tf calls.
Running the same example in graph mode using the tf.function decorator we
also get the graph placement appended to the end of the log.
x: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0
Sqrt: (Sqrt): /job:localhost/replica:0/task:0/device:GPU:3
Identity: (Identity): /job:localhost/replica:0/task:0/device:GPU:3
identity_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:3
As seen above the tensor operations were deployed on GPU 3.
Beware of the optimiser! Optimisation may alter your graphs!
The TensorFlow optimiser Grappler performs optimisations during graph execution which might result in some of your logged operations not being executed.
Read more about Grappler in the official documentation.
If you have discovered, or been notified, that your jobs are not efficient, we recommend you profile your application. Read our profiling guide for tips and documentation of available tools.
Eager vs. graph execution🔗
TensorFlow for most long-running and non-trivial training cases works by
constructing and executing graphs. This makes the well-known debugging
techniques (such as printing or using a step debugger) not very useful as you
are most likely interested in actual values during execution rather than their
symbolic representation, which is what you will see if you use e.g. python
print or attach pdb.
Tensor(“MatMul:0”, shape = (2,2), dtype = float32);
The above symbolic representation of a 2x2 float32 tensor from a
matmul-operation is what a normal Python print would produce. In eager
execution the output would include the values as well:
tf.Tensor( [ [ 1337.0 1.0] [ 1.0 1337.0] ], shape= (2,2), dtype = floatt32)
So - we conclude that TensorFlow supports two different types of execution: eager and graph. Eager execution is basically operation-by-operation execution, as you would expect from working with any imperative coding language, while graph execution (that may take place in native code) works by constructing (known as tracing) and executing a graph. Eager execution is in general (but not always) slower but easier to debug compared to graph execution, hence the saying code with eager execution, run with graphs. As of TensorFlow 2.0 the default execution is eager, but you may find it useful to switch between the two modes during debugging. It only requires a few changes to the code.
Assume the following very simple function sqrt.
import tensorflow as tf
def sqrt(x):
return tf.math.sqrt(x)
x = tf.constant([1.0, 2.0, 3.0])
x_sqrt = sqrt(x)
If you run this in TensorFlow 2.0 it will run in eager mode. The same code in graph mode would look like:
import tensorflow as tf
def sqrt(x):
return tf.math.sqrt(x)
x = tf.constant([1.0, 2.0, 3.0])
x_sqrt = sqrt(x)
tf_sqrt = tf.function(sqrt)
The only difference is that we wrapped sqrt in a tf.function call. It is
also common to see functions wrapped using the tf.function decorator.
import tensorflow as tf
@tf.function
def sqrt(x):
return tf.math.sqrt(x)
A TensorFlow function runs the decoration function, when possible, as a TensorFlow graph. This has the side effect that any print statements included in the function would only get executed once as the graph is constructed.
Read more about tf.function in the
official documentation.
You can turn all supported calls to tf.graph into eager mode by setting
tf.config.run_functions_eagerly(True). In our simple example above there is
little benefit, but for a function that iterates it can be useful to print (or
checkpoint) the intermediate values.
Printing with tf.print🔗
You should substitute Python print with tf.print to print the actual
Tensor values during graph execution. Internally tf.print works by appending
a pair of nodes (String and PrintV2) to graph. As in traditional debugging,
adding prints may work if the code is small or you already have gathered
enough information to know roughly where the problem is. Adding tf.print to
every tensor in your code is not effective.
@tf.function
def my_func(x):
for i in range(x):
intermediate = tf.math.reduce_sum(x) + x[i] + pi
print(intermediate) # Symbolic representation only (printed during tracing!)
tf.print(intermediate) # Prints the actual tensor values
Read more about tf.print in the
official documentation.
Check for NaN and +-Infinity🔗
TensorFlow 2.4.1 introduced tf.debugging.enable_check_numerics that when
activated will stop execution as soon as a tensor contains Infinity or NaN.
The call introduces numerical checks in your graph - overhead - and thereby
extending your runtime and memory requirements. It should only be set during
debugging.
This contrived example illustrates how a +Inf is caught and reported.
import tensorflow as tf
tf.debugging.enable_check_numerics(False) # Numeric checking is disabled
@tf.function
def weighted_normalized(vec):
eps = tf.constant(0.005)
res = vec / (tf.math.reduce_sum(vec) + eps)
return res
# This is fine.
vec = tf.constant([0.0, 1.0, 2.0, 3.0])
wn_vec = weighted_normalized(vec) # [0 0.166527897 0.333055794 0.499583662])
# This is not fine... but with numeric checking disabled it will not be caught.
vec2 = tf.constant([-0.005, 0])
wn_vec2 = weighted_normalized(vec2) # [-inf nan] <-- NOT OK
If we run the above we get no indications that something is wrong even though
the second tensor contains [-inf nan]. If we re-run with
tf.debugging.enable_check_numerics(True) we get the following printed to stdout.
Traceback (most recent call last):
File "test_tf.check_numerics", line 22, in <module>
wn_vec2 = weighted_normalized(vec2) # [-inf nan] <-- NOT OK
File "/apps/Alvis/software/MPI/GCC-CUDA/9.3.0-11.0.2/OpenMPI/4.0.3/TensorFlow/2.3.1-Python-3.8.2/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
result = self._call(*args, **kwds)
File "/apps/Alvis/software/MPI/GCC-CUDA/9.3.0-11.0.2/OpenMPI/4.0.3/TensorFlow/2.3.1-Python-3.8.2/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 814, in _call
results = self._stateful_fn(*args, **kwds)
File "/apps/Alvis/software/MPI/GCC-CUDA/9.3.0-11.0.2/OpenMPI/4.0.3/TensorFlow/2.3.1-Python-3.8.2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "/apps/Alvis/software/MPI/GCC-CUDA/9.3.0-11.0.2/OpenMPI/4.0.3/TensorFlow/2.3.1-Python-3.8.2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1843, in _filtered_call
return self._call_flat(
File "/apps/Alvis/software/MPI/GCC-CUDA/9.3.0-11.0.2/OpenMPI/4.0.3/TensorFlow/2.3.1-Python-3.8.2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1923, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "/apps/Alvis/software/MPI/GCC-CUDA/9.3.0-11.0.2/OpenMPI/4.0.3/TensorFlow/2.3.1-Python-3.8.2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 545, in call
outputs = execute.execute(
File "/apps/Alvis/software/MPI/GCC-CUDA/9.3.0-11.0.2/OpenMPI/4.0.3/TensorFlow/2.3.1-Python-3.8.2/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 138, in execute_with_callbacks
tensors = quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
File "/apps/Alvis/software/MPI/GCC-CUDA/9.3.0-11.0.2/OpenMPI/4.0.3/TensorFlow/2.3.1-Python-3.8.2/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:
!!! Detected Infinity or NaN in output 0 of graph op "RealDiv" (# of outputs: 1) !!!
dtype: <dtype: 'float32'>
shape: (2,)
Input tensors (2):
0: Tensor("vec:0", shape=(2,), dtype=float32)
1: Tensor("add:0", shape=(), dtype=float32)
Graph name: "weighted_normalized"
Stack trace of op's creation ("->": inferred user code):
+ ... (Omitted 18 frames)
+ ...8/site-packages/tensorflow/python/framework/ops.py (L1949) __init__
| self._traceback = tf_stack.extract_stack()
: Tensor had -Inf and NaN values
[[node truediv/CheckNumericsV2 (defined at test_tf.check_numerics:10) ]] [Op:__inference_weighted_normalized_43]
Function call stack:
weighted_normalized
Which clearly says that we have numerical issues in the graph weighted_normalized.
Read more about tf.debugging.enable_check_numerics in the
official documentation
Increase logging🔗
Logfiles can be a rich source of information, and every now and then they contain just enough to point us in the right direction. TensorFlow, together with Keras, has many types of logs, for instance, autograph logs, model loss logs, summary usage logs, training metric logs etc. In general most logging facilities support different levels of severity (sometimes called levels). TensorFlow supports the following five standard severity levels, in order of least to most severe: Supported levels: * DEBUG * ERROR * FATAL * INFO * WARN
The most verbose setting is the least severe - i.e DEBUG level produces the
most output, and most will likely not be relevant.
The following example gets the TensorFlow log and sets it in the DEBUG level.
import tensorflow as tf
import logging
tf_logger = tf.get_logger("tensorflow") # Any name will work
tf_logger.setLevel(tf_logger.DEBUG)
Warning: Beware of large logs and save them in a NOBACKUP area or in tmpdir.
Read more about tf.get_logger in the
official documentation.
Checkpoint your code🔗
Long running cluster jobs that for whatever reason crashes is not only very
annoying but also wastes precious compute hours from your allocation. In the
case of TensorFlow crashes can be challenging to debug, especially if it is
due to numerical instabilities (although there are tricks to discover NaN and
+-Inf, as we will show) that appear far along in training. Checkpointing
allows you to regularly and automatically save the state of your model to disk.
If a crash occurs, you can debug or resume your job at the most recent (and
correct) checkpoint. It is thus recommended for several reasons that you
investigate how checkpointing fits your code - and TensorFlow offers a few options.
You should start by looking into tf.train.Checkpoint or the Keras callback
tf.keras.callbacks.ModelCheckpoint. If you checkpoint every hour, you can
begin your debugging at the latest one hour before the bug (or crash) began.
Here is an example
how checkpointing works in TensorFlow on the Alvis system.
Read more about checkpointing in the official documentation.
Visualise using TensorBoard🔗
TensorBoard is a visualisation toolkit that comes bundled with TensorFlow. TensorBoard can be used for many different types of debugging, and not exclusively for TensorFlow. For instance, you can visualise the resulting graph, find traces and profiling information.
.
TensorBoard is interactive and you can inspect your graph in a browser. For more information about TensorBoard read our TensorBoard guide.
Experimental API debugging🔗
We finish with a recommendation to keep an eye in the experimental API for new features related to debugging. Note that the API that is marked as experimental may change with short notice, and may become dropped in later versions.
Warning: tf.debugging.experimental.enable_dump_debug_info as of TensorFlow
2.3 is marked as experimental. How you call it may change between versions.
Please consult the
documentation for tf.debugging
for the current way of using it for the version of TensorFlow you are running.
You can dump large amount of debug information to a directory using the
tf.debugging.experimental.enable_dump_debug_info() call at the start of your
program.
import tensorflow as tf
tf.debugging.experimental.enable_dump_debug_info(
dump_root="logs/dbg",
tensor_debug_mode="FULL_HEALTH",
circular_buffer_size=-1)
# Rest of program
The option tensor_debug_mode in the above example is set to maximum
verbosity, producing information about each floating-dtype tensor including
counts of +- Inf and NaN. You can increase the amount of debug information
by increasing the circular_buffer_size. The value -1 disable the circular
buffer - and we keep all data. The debug data also includes events and
annotated source code tracing data. It is mean to be visualised using
TensorBoard.
You view the data by launching TensorBoard on the same machine were you dumped
the data. TensorBoard is bundled with TensorFlow. If you have loaded TensorFlow
from the module system you can launch TensorBoard using the tensorboard
command.
$ tensorboard --logdir logs/dbg
Read more about the experimental API in the official documentation
Reading datasets🔗
Machine learning datasets commonly can be made up from a multitude of files. In HPC environments these can be less than ideal. Reading (and writing) is generally a lot slower compared when done to multiple files compared to a single large files.
To get around this you should read directly from archives or if you are doing some preprocessing and are planning on using your data multiple, then you can save the preprocessed data into a few tfrecords instead.
To read directly from archives you can use the TensorFlow Datasets module
(module spider TensorFlow-Datasets). For example to read images:
import tensorflow_datasets as tfds
# Initialize to get access to method (we do not plan on downloading anything)
dl_manager = tfds.download.DownloadManager(download_dir='/tmp')
# Iterate over archive
path_to_archive = '/mimer/NOBACKUP/Datasets/Lyft-level5/Perception/train.tar'
image_feature = tfds.core.features.Image()
for filename, f in dl_manager.iter_archive(path_to_archive):
if filename.endswith('.jpeg'):
img = image_feature.decode_example(f.read())
print(img.shape)