Dyverso Crashes when using OpenCL-GPU or CUDA

[ChristianZ]

If you are looking for a well-working Dyverso scene with crown daemon, load the "basic_crown" scene from the "dyverso" example scenes that come with RF10.

In the meantime, I was playing around a little more with different Dyverso simulations with and without Cuda, but I am no longer sure if it is the GPU that causing the crash, because when it did crash, it usually happened after I added a new element to the scene, reset and run again. After the crash the same scene often worked again after restarting RF10...
So it happened with this scene
https://www.dropbox.com/s/e73w2rr3uluxa ... r.zip?dl=0
that produced this crash report
https://www.dropbox.com/s/l6j4b39v8bny5 ... f.zip?dl=0

[ivalduan]

Ivalduan,

Here are the details about this crash problem:

1 - Using Dyverso square emitter, 1 Crown daemon in scene (please see image attached).
2 - Using Windows 10 Pro Edition, 32 GB Ram, 512 SSD Disk, Asus Geforce GTX 1080 Graphics Card.
3 - As soon as I edited internal and external pressure on the Dyverso Domain, hit Reset and Simulate the crash occurs. but it will occur also editing other parameters besides those ones.

I am attaching for you to analyse:

a) a screen shot of the crash, here:
https://drive.google.com/open?id=0B_mgF ... nZYYWRmVHM

b) a compressed file of the scene, here:
https://drive.google.com/open?id=0B_mgF ... mw1U2VQRUE

I really hope you can help in some way. It is so important

Thank you so much.
Rui

Hello RiuManuel,

The problem with your scene & CUDA seems to be the "Internal pressure" being equal to 0.0, once you make it > 0.0 it works. We will fix the valid range of this parameter for the next update. IMHO it makes no sense for a liquid to avoid pressure at all, default 1.0 should work 95% of the times. If you want particles without interaction between them and volume preservation you can use the dumb material.

In addition to that, the scene has no gravity so particles fly away, and the emitter is producing particles inside scene objects. Be careful with this, the solver will correct suddenly the particles inside objects producing violent results.

I made some tweaks, the crown acts at frame 30. Hope it helps.

tst3_mod.zip

(134.18 KiB) Downloaded 49 times

tst3_mod.png (127.38 KiB) Viewed 1317 times

[ivalduan]

It works but keeps crashing in Cuda mode.

For some reason it seems it didnt keep your project settings: the dyverso liquid explodes and the generated Crown is too thick, no matter what is in the With parameter.

I'm doomed.

Can't reproduce a CUDA crash with this one. Thanks ChristianZ.

tst4.png (238.69 KiB) Viewed 1316 times

[ivalduan]

If you are looking for a well-working Dyverso scene with crown daemon, load the "basic_crown" scene from the "dyverso" example scenes that come with RF10.

In the meantime, I was playing around a little more with different Dyverso simulations with and without Cuda, but I am no longer sure if it is the GPU that causing the crash, because when it did crash, it usually happened after I added a new element to the scene, reset and run again. After the crash the same scene often worked again after restarting RF10...
So it happened with this scene
https://www.dropbox.com/s/e73w2rr3uluxa ... r.zip?dl=0
that produced this crash report
https://www.dropbox.com/s/l6j4b39v8bny5 ... f.zip?dl=0

Thanks for the report ChristianZ,

I will try here similar workflows, if you discover the exact chain of steps that achieves the crash let me know .

[RuiManuel]

Thank you for taking the time for analyzing my problem in such a detail.

I'm gonna read all this carefully and test the parameter values.

Rui.

[ChristianZ]

Today I had an unexpected exit of RF10 in terminal mode with a scene using Dyverso (SPH-particles, high viscosity) and Cuda. At least we can say that the error message made clear what the primary issue at this point was:

>14:37:31: ================ Frame finished ================

>14:37:31: ================ Frame 75 started ================
....................................................CUDA ERROR: out of memory cudaMalloc 712
CUDA ERROR: out of memory cudaMalloc 712
CUDA ERROR: out of memory cudaMalloc 712
..........libc++abi.dylib: terminating with uncaught exception of type tbb::captured_exception: after cub_::DeviceRadixSort::SortPairs(1): out of memory
/var/folders/pt/11n407748xj1gh0059_xkkv00000gn/T/Zviscous.CPm419.command: line 1: 3163 Abort trap: 6 "/Applications/RealFlow 10/RealFlow.app/Contents/MacOS/RealFlow" -nogui -range 0 200 -threads 16 "/Volumes/Seagate/RF-scenes-seagate/Zviscous/Zviscous.flw"
logout

On this MacPro I am using a Nvidia GTX970 with 4GB RAM.

[Forester]

I'm getting the same kinds of crashes using Dyverso with CUDA invoked. Any change at all in the scene, or any attempt to add a Filter Deamon or any attempt to add a Sim Script (Python) causes it to crash. Approximately the same error message as you are receiving. I will test this more thoroughly and file a bug report, also.
Darn! Same phenomena with OpenCL.

[ivalduan]

Today I had an unexpected exit of RF10 in terminal mode with a scene using Dyverso (SPH-particles, high viscosity) and Cuda. At least we can say that the error message made clear what the primary issue at this point was:

>14:37:31: ================ Frame finished ================

>14:37:31: ================ Frame 75 started ================
....................................................CUDA ERROR: out of memory cudaMalloc 712
CUDA ERROR: out of memory cudaMalloc 712
CUDA ERROR: out of memory cudaMalloc 712
..........libc++abi.dylib: terminating with uncaught exception of type tbb::captured_exception: after cub_::DeviceRadixSort::SortPairs(1): out of memory
/var/folders/pt/11n407748xj1gh0059_xkkv00000gn/T/Zviscous.CPm419.command: line 1: 3163 Abort trap: 6 "/Applications/RealFlow 10/RealFlow.app/Contents/MacOS/RealFlow" -nogui -range 0 200 -threads 16 "/Volumes/Seagate/RF-scenes-seagate/Zviscous/Zviscous.flw"
logout

On this MacPro I am using a Nvidia GTX970 with 4GB RAM.

Hello Christian,
yep, it seems like your GPU runned out of memory for that simulation.
We are considering adding a fallback to resume the simulation on CPU when this happens.

A bit of curiosity, how many particles were on the scene when the crash happened?

[ivalduan]

I'm getting the same kinds of crashes using Dyverso with CUDA invoked. Any change at all in the scene, or any attempt to add a Filter Deamon or any attempt to add a Sim Script (Python) causes it to crash. Approximately the same error message as you are receiving. I will test this more thoroughly and file a bug report, also.
Darn! Same phenomena with OpenCL.

Hello Forester,

One of the bugs solved for the coming update is titled "Several issues related to daemons adding particle data (sheeter, crown, magic) after a Dyverso GPU simulation has taken place." I assume this is related to your problem.
People claimed for RF2015 that simulations took some annoying time to start while all the GPU stuff was being initialized. To avoid that in RF10 all GPU stuff is maintained always ON after being initialized the first time to improve responsiveness, these bugs come from that fact.

Maybe you can try to disable the GPU checkbox while you are adding or removing Dyverso nodes (like a filter) to the scene; then enabling it after to simulate, until the update arrives. Crash should dissapear.

[ChristianZ]

A bit of curiosity, how many particles were on the scene when the crash happened?

Sorry, I don't know exactly because I have deleted that project already, but it was more than 3 millions with that card with 4GB RAM.