View Issue Details

IDProjectCategoryView StatusLast Update
0004572Slicer4Module VolumeRenderingpublic2018-06-16 10:15
ReportermuratmagaAssigned Tofinetjul 
PrioritynormalSeveritymajorReproducibilityalways
Status assignedResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0004572: Volume rendering gives TDR error on windows 10 with quadro K4100M
Description

4.9.0-2018-06-11 r27246 on windows 10 crashes with a dataset that works fine on 4.8.1. This is with regular volume rendering, not with experimental option. GPU rendering is set to 4GB (which is the memory this card has). Happens everytime. This is on a laptop with i7 chip + Quadro K4100M

Sample dataset
https://app.box.com/s/imcvz02opq5odos0cowto86p4piz392a

Steps To Reproduce

Load the nifti file from the link. Switch to 4 up view. Switch to VR, enable rendering, slide the transfer function down to about 5, center and zoom into the specimen. It sutters and crashes.

Same steps doesn't cause a crash in 4.8.1

TagsNo tags attached.

Activities

muratmaga

muratmaga

2018-06-13 19:00

reporter   ~0015891

Under the prefenceses GPU rendering is selected for volume rendering options, plus 4GB GPU RAM is specified. However, when I switch to volume rendering module and look at the advanced tag, gpu ram shows as 0MB. I manually switched it to 4GB prior to rendering. Made no difference. Still crashed.

I made sure slicer-real-app.exe is added to the nvidia control panel to make sure it is using the nvidia chipset. While

lassoan

lassoan

2018-06-15 13:25

developer   ~0015892

Quadro cards often have compatibility problems (probably because so few people use them and report errors), so if you have the choice, switch to GeForce (you get much better performance for the same money anyways).

It may still be a Slicer issue though. Could you try a nightly build from a month ago?
https://download.slicer.org/?offset=-30

Also, If your GPU has 4GB RAM then don't allow the volume renderer use 4GB. Try 1-2GB instead.

muratmaga

muratmaga

2018-06-15 13:29

reporter   ~0015893

It is a laptop, I don't have a choice to change it. Same dataset works perfectly fine with stable version with specified 4GB GPU RAM on the same system. I will give it a try with an older nightly.

lassoan

lassoan

2018-06-15 14:01

developer   ~0015894

Latest stable version of Slicer used VTK7 and its ancient OpenGL API, which had good hardware compatibility (ran on 15-year-old hardware). However, VTK switched to more recent OpenGL APIs, which offer much better performance but there are more potential compatibility issues, as they use more enhanced features.

Is your BIOS and graphics card driver up-to-date? Does volume rendering in ParaView work for you? Does CPU volume rendering in Slicer work?

muratmaga

muratmaga

2018-06-15 14:12

reporter   ~0015896

It is a laptop from 2014 (dell precision M6800), bios is the latest for it. I will try updating the Nvidia driver, though I think the existing driver is only a 6 months old or so (v390.77). CPU rendering works fine (but slow). I don't have paraview.

lassoan

lassoan

2018-06-15 14:18

developer   ~0015897

Last edited: 2018-06-15 14:19

View 2 revisions

You can download paraview from here: https://www.paraview.org/download/. It uses the same rendering backend as Slicer. If it crashes, too then the error is most likely in VTK or lower level.

In the future, I would recommend to stay away from Quadro cards, unless you only use applications that are certified to be compatible with them. Quadro cards are optimized to work with a list of 100 or so "professional" applications and often not work very well with others.

pinter

pinter

2018-06-15 15:12

developer   ~0015898

I tried to reproduce, and although it gets slow sometimes, there is no crash. I have a GeForce 1060 3GB.
What are your settings? VTK GPU Ray Casting I assume. What about Quality (Adaptive/Normal/Maximum)? Did you use any transfer function other than the default? What does it mean "slide the transfer function down to about 5"?

muratmaga

muratmaga

2018-06-15 15:55

reporter   ~0015899

I reproduced it on a desktop with TITAN (original one with 6GB) on windows 7. It has the latest set of drivers from NVIDIA.

I use VTK GPU ray casting with adaptive quality (8fp). I modified the default transfer function such that the low point on scalar opacity function is to 5 and the high is set to 6 (so a very short ramp). Then try interacting with the volume extensively zoom in, zoom out, spin etc. It works about 5-10 secs, and then driver crashes (on Windows 7) or get a TDR (on windows 10).

pinter

pinter

2018-06-15 16:56

developer   ~0015900

Thanks for the details! When I did what you suggested Slicer just quit. It wasn't even a crash. I'm not sure what could be possibly done to diagnose the exact reason.

muratmaga

muratmaga

2018-06-15 17:01

reporter   ~0015901

On windows I do get an error, but not from Slicer from Nvidia driver (both on Windows 7 and 10). If you are on Linux/mac may be syslogs may help.
Again same dataset works perfectly fine on 4.8.1 with the same settings.

lassoan

lassoan

2018-06-16 10:04

developer   ~0015902

Last edited: 2018-06-16 10:04

View 2 revisions

I could reproduce the crash, too. Details were described in the system logs: it was a TDR (http://nvidia.custhelp.com/app/answers/detail/a_id/3633, https://docs.microsoft.com/en-us/windows-hardware/drivers/display/timeout-detection-and-recovery). It seems that the GPU is simply too busy because of the huge workload.

I've "fixed" the issue by increasing TDR delay to 10 seconds (by adding TdrDelay=10 to registry https://docs.microsoft.com/en-us/windows-hardware/drivers/display/tdr-registry-keys) and setting the image's data type to unsigned char, and using Adaptive or Normal setting. I could still get TDR on maximum quality.

While searching for solutions for TDR error on the web, I've found that many scientific software has this error and the recommended solution is to increase TDR delay, so this seems to be a common solution.

However, I think the GPU raycast mapper could be improved to reduce the chance of this error.

  1. When setting GPU memory usage limit to a lower value then the volume supposed to be downscaled in the mapper to fit in that size. I don't see an effect of maximum GPU memory size, so maybe this mechanism is broken in Slicer or VTK.

  2. vtkGPUVolumeRayCastMapper could be changed to perform processing in smaller batches. However, implementing this is probably not trivial, it could slow down rendering, should not be necessary for commonly occurring volume sizes, and there are multiple workarounds (increase TDR delay or downsample volume), so I don't think this will be very high on the priority list of VTK developers. We should still report this, so that developers are aware of this and users who encounter this issue get a clue about what is happening and what can be done.
lassoan

lassoan

2018-06-16 10:15

developer   ~0015903

Entered a VTK issue: https://gitlab.kitware.com/vtk/vtk/issues/17342

Issue History

Date Modified Username Field Change
2018-06-13 18:51 muratmaga New Issue
2018-06-13 18:51 muratmaga Status new => assigned
2018-06-13 18:51 muratmaga Assigned To => finetjul
2018-06-13 18:52 muratmaga Summary Volume rendering gives TDR error on windows 10 with quadro 2200M => Volume rendering gives TDR error on windows 10 with quadro K4100M
2018-06-13 19:00 muratmaga Note Added: 0015891
2018-06-15 13:25 lassoan Note Added: 0015892
2018-06-15 13:29 muratmaga Note Added: 0015893
2018-06-15 14:01 lassoan Note Added: 0015894
2018-06-15 14:12 muratmaga Note Added: 0015896
2018-06-15 14:18 lassoan Note Added: 0015897
2018-06-15 14:19 lassoan Note Edited: 0015897 View Revisions
2018-06-15 15:12 pinter Note Added: 0015898
2018-06-15 15:55 muratmaga Note Added: 0015899
2018-06-15 16:56 pinter Note Added: 0015900
2018-06-15 17:01 muratmaga Note Added: 0015901
2018-06-16 10:04 lassoan Note Added: 0015902
2018-06-16 10:04 lassoan Note Edited: 0015902 View Revisions
2018-06-16 10:15 lassoan Note Added: 0015903