## Release Notes

### Nsight Systems 2020.3.2

#### What's New

- OpenMP trace is now available for Linux on x86 and IBM Power targets.

- IBM Power target now has support for OS runtime trace.

- IBM Power target now supports backtraces using DWARF's CFI (Call Frame Information).

- Various bug fixes and performance enhancements.

#### Known Issues

##### General Issues

- Profiling greater than 5 minutes is not officially supported at this time. Profiling high activity applications, on high performance machines, over a long analysis time can create large result files that may take a very long time to load, run out of memory, or lock up the system. If you have a complex application, we recommend starting with a short profiling session duration of no more than 5 minutes for your initial profile. If your application has a natural repeating pattern, often referred to as a frame, you may typically only need a few of these. This suggested limit will increase in future releases.

- Attaching or re-attaching to a process from the GUI is not supported with the x86_64 Linux target. Equivalent results can be obtained by using the interactive CLI to launch the process and then starting and stopping analysis at multiple points.

- The CLI uses default options when tracing APIs. See respective library portion of this documentation for information on defaults. This limitation will be removed in a future version of the product.

- There is an upper bound on the default size used by the tool to record trace events during the collection. If you see the following diagnostic error, then Nsight Systems hit the upper limit.

  >     Reached the size limit on recording trace events for this process. Try
  >     reducing the profiling duration or reduce the number of features traced.

- When profiling a framework or application that uses CUPTI, like some versions of TensorFlow(tm), Nsight Systems will not be able to trace CUDA usage due to limitations in CUPTI. These limitations will be corrected in a future version of CUPTI. Consider turning off the application's use of CUPTI if CUDA tracing is required.

  As an example, in the TensorFlow `mnist_with_summaries.py` tutorial, you will be able to use Nsight Systems to perform CUDA trace if you remove usage of RunOptions.FULL_TRACE from the code. See RunOptions documentation.

- Tracing an application that uses a non-thread safe memory allocator is not supported.

- Tracing OS Runtime libraries in an application that preloads glibc symbols is unsupported and can lead to undefined behavior.

- Nsight Systems cannot profile applications launched through a virtual window manager like GNU Screen.

##### Docker Issues

- In a Docker, when a system's host utilizes a kernel older than v4.3, it is not possible for Nsight Systems to collect sampling data unless both the host and Docker are running a RHEL or CentOS operating system utilizing kernel version 3.10.1-693 or newer. A user override for this will be made available in a future version.

- When `docker exec` is called on a running container and stdout is kept open from a command invoked inside that shell, the exec shell hangs until the command exits. You can avoid this issue by running with `docker exec --tty`. See the bug reports at:

  https://github.com/moby/moby/issues/33039

  https://github.com/drud/ddev/issues/732

##### CUDA Trace Issues

- If the target application uses multiple streams from multiple threads, CUDA event buffers may not be released properly. In this case the you will see the following diagnostic error:

  >     Couldn't allocate CUPTI bufer x times. Some CUPTI events may be missing.

  Please contact the Nsight Systems team.

- CUDA GPU trace collection requires a fraction of GPU memory. If your application utilizes all available GPU memory, CUDA trace might not work or can break your application. As an example cuDNN application can crash with `CUDNN_STATUS_INTERNAL_ERROR` error if GPU memory allocation fails.

- For older Linux kernels, prior to 4.4, when profiling very short-lived applications (~1 second) that exit in the middle of the profiling session, it is possible that Nsight Systems will not show the CUDA events on the timeline.

- When more than 64k serialized CUDA kernels and memory copies are executed in the application, you may encounter the following exception during profiling:

  >     InvalidArgumentException: "Wrong event order detected"

  Please upgrade to the CUDA 9.2 driver at minimum to avoid this problem. If you cannot upgrade, you can get a partial analysis, missing potentially a large fraction of CUDA events, by using the CLI.

Copyright (c) 2012-2020, NVIDIA Corporation. All rights reserved.