Dyverso Crashes when using OpenCL-GPU or CUDA

Describe your realflow bugs
User avatar
ChristianZ
Posts: 3267
Joined: Sat Sep 09, 2006 11:37 am
Location: CH

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby ChristianZ » Sun Feb 12, 2017 6:01 pm

If you are looking for a well-working Dyverso scene with crown daemon, load the "basic_crown" scene from the "dyverso" example scenes that come with RF10.

In the meantime, I was playing around a little more with different Dyverso simulations with and without Cuda, but I am no longer sure if it is the GPU that causing the crash, because when it did crash, it usually happened after I added a new element to the scene, reset and run again. After the crash the same scene often worked again after restarting RF10...
So it happened with this scene
https://www.dropbox.com/s/e73w2rr3uluxa ... r.zip?dl=0
that produced this crash report
https://www.dropbox.com/s/l6j4b39v8bny5 ... f.zip?dl=0
RF10 standard, System specs: OSX 10.11, MacPro 4.1 2xXeon, GTX980Ti, 64 GB RAM


ivalduan
Posts: 13
Joined: Fri Feb 10, 2017 10:11 am

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby ivalduan » Mon Feb 13, 2017 5:21 pm

RuiManuel wrote:Ivalduan,

Here are the details about this crash problem:

1 - Using Dyverso square emitter, 1 Crown daemon in scene (please see image attached).
2 - Using Windows 10 Pro Edition, 32 GB Ram, 512 SSD Disk, Asus Geforce GTX 1080 Graphics Card.
3 - As soon as I edited internal and external pressure on the Dyverso Domain, hit Reset and Simulate the crash occurs. but it will occur also editing other parameters besides those ones.

I am attaching for you to analyse:

a) a screen shot of the crash, here:
https://drive.google.com/open?id=0B_mgF ... nZYYWRmVHM

b) a compressed file of the scene, here:
https://drive.google.com/open?id=0B_mgF ... mw1U2VQRUE

I really hope you can help in some way. It is so important :(

Thank you so much.
Rui


Hello RiuManuel,

The problem with your scene & CUDA seems to be the "Internal pressure" being equal to 0.0, once you make it > 0.0 it works. We will fix the valid range of this parameter for the next update. IMHO it makes no sense for a liquid to avoid pressure at all, default 1.0 should work 95% of the times. If you want particles without interaction between them and volume preservation you can use the dumb material.

In addition to that, the scene has no gravity so particles fly away, and the emitter is producing particles inside scene objects. Be careful with this, the solver will correct suddenly the particles inside objects producing violent results.

I made some tweaks, the crown acts at frame 30. Hope it helps.
tst3_mod.zip
(134.18 KiB) Downloaded 27 times

tst3_mod.png
tst3_mod.png (127.38 KiB) Viewed 819 times

ivalduan
Posts: 13
Joined: Fri Feb 10, 2017 10:11 am

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby ivalduan » Mon Feb 13, 2017 5:38 pm

RuiManuel wrote:It works but keeps crashing in Cuda mode.

For some reason it seems it didnt keep your project settings: the dyverso liquid explodes and the generated Crown is too thick, no matter what is in the With parameter.

I'm doomed. :(


Can't reproduce a CUDA crash with this one. Thanks ChristianZ.
tst4.png
tst4.png (238.69 KiB) Viewed 818 times

ivalduan
Posts: 13
Joined: Fri Feb 10, 2017 10:11 am

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby ivalduan » Mon Feb 13, 2017 6:34 pm

ChristianZ wrote:If you are looking for a well-working Dyverso scene with crown daemon, load the "basic_crown" scene from the "dyverso" example scenes that come with RF10.

In the meantime, I was playing around a little more with different Dyverso simulations with and without Cuda, but I am no longer sure if it is the GPU that causing the crash, because when it did crash, it usually happened after I added a new element to the scene, reset and run again. After the crash the same scene often worked again after restarting RF10...
So it happened with this scene
https://www.dropbox.com/s/e73w2rr3uluxa ... r.zip?dl=0
that produced this crash report
https://www.dropbox.com/s/l6j4b39v8bny5 ... f.zip?dl=0


Thanks for the report ChristianZ,

I will try here similar workflows, if you discover the exact chain of steps that achieves the crash let me know :oops: .

RuiManuel
Posts: 7
Joined: Fri Feb 10, 2017 1:08 am

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby RuiManuel » Tue Feb 14, 2017 9:52 pm

Thank you for taking the time for analyzing my problem in such a detail.

I'm gonna read all this carefully and test the parameter values.

Rui.

User avatar
ChristianZ
Posts: 3267
Joined: Sat Sep 09, 2006 11:37 am
Location: CH

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby ChristianZ » Fri Feb 17, 2017 1:43 pm

Today I had an unexpected exit of RF10 in terminal mode with a scene using Dyverso (SPH-particles, high viscosity) and Cuda. At least we can say that the error message made clear what the primary issue at this point was:
>14:37:31: ================ Frame finished ================

>14:37:31: ================ Frame 75 started ================
....................................................CUDA ERROR: out of memory cudaMalloc 712
CUDA ERROR: out of memory cudaMalloc 712
CUDA ERROR: out of memory cudaMalloc 712
..........libc++abi.dylib: terminating with uncaught exception of type tbb::captured_exception: after cub_::DeviceRadixSort::SortPairs(1): out of memory
/var/folders/pt/11n407748xj1gh0059_xkkv00000gn/T/Zviscous.CPm419.command: line 1: 3163 Abort trap: 6 "/Applications/RealFlow 10/RealFlow.app/Contents/MacOS/RealFlow" -nogui -range 0 200 -threads 16 "/Volumes/Seagate/RF-scenes-seagate/Zviscous/Zviscous.flw"
logout

On this MacPro I am using a Nvidia GTX970 with 4GB RAM.
RF10 standard, System specs: OSX 10.11, MacPro 4.1 2xXeon, GTX980Ti, 64 GB RAM

Forester
Posts: 86
Joined: Mon Mar 15, 2010 4:36 am
Contact:

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby Forester » Sat Feb 18, 2017 8:42 pm

I'm getting the same kinds of crashes using Dyverso with CUDA invoked. Any change at all in the scene, or any attempt to add a Filter Deamon or any attempt to add a Sim Script (Python) causes it to crash. Approximately the same error message as you are receiving. I will test this more thoroughly and file a bug report, also.
Darn! Same phenomena with OpenCL.
Windows 7 Pro 64 - Realflow 10.1.2.0162 (Nov 2017 patched) Standard Version

ivalduan
Posts: 13
Joined: Fri Feb 10, 2017 10:11 am

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby ivalduan » Tue Feb 21, 2017 2:08 pm

ChristianZ wrote:Today I had an unexpected exit of RF10 in terminal mode with a scene using Dyverso (SPH-particles, high viscosity) and Cuda. At least we can say that the error message made clear what the primary issue at this point was:
>14:37:31: ================ Frame finished ================

>14:37:31: ================ Frame 75 started ================
....................................................CUDA ERROR: out of memory cudaMalloc 712
CUDA ERROR: out of memory cudaMalloc 712
CUDA ERROR: out of memory cudaMalloc 712
..........libc++abi.dylib: terminating with uncaught exception of type tbb::captured_exception: after cub_::DeviceRadixSort::SortPairs(1): out of memory
/var/folders/pt/11n407748xj1gh0059_xkkv00000gn/T/Zviscous.CPm419.command: line 1: 3163 Abort trap: 6 "/Applications/RealFlow 10/RealFlow.app/Contents/MacOS/RealFlow" -nogui -range 0 200 -threads 16 "/Volumes/Seagate/RF-scenes-seagate/Zviscous/Zviscous.flw"
logout

On this MacPro I am using a Nvidia GTX970 with 4GB RAM.


Hello Christian,
yep, it seems like your GPU runned out of memory for that simulation.
We are considering adding a fallback to resume the simulation on CPU when this happens.

A bit of curiosity, how many particles were on the scene when the crash happened?

ivalduan
Posts: 13
Joined: Fri Feb 10, 2017 10:11 am

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby ivalduan » Tue Feb 21, 2017 2:26 pm

Forester wrote:I'm getting the same kinds of crashes using Dyverso with CUDA invoked. Any change at all in the scene, or any attempt to add a Filter Deamon or any attempt to add a Sim Script (Python) causes it to crash. Approximately the same error message as you are receiving. I will test this more thoroughly and file a bug report, also.
Darn! Same phenomena with OpenCL.


Hello Forester,

One of the bugs solved for the coming update is titled "Several issues related to daemons adding particle data (sheeter, crown, magic) after a Dyverso GPU simulation has taken place." I assume this is related to your problem.
People claimed for RF2015 that simulations took some annoying time to start while all the GPU stuff was being initialized. To avoid that in RF10 all GPU stuff is maintained always ON after being initialized the first time to improve responsiveness, these bugs come from that fact.

Maybe you can try to disable the GPU checkbox while you are adding or removing Dyverso nodes (like a filter) to the scene; then enabling it after to simulate, until the update arrives. Crash should dissapear.

User avatar
ChristianZ
Posts: 3267
Joined: Sat Sep 09, 2006 11:37 am
Location: CH

Re: Dyverso Crashes when using OpenCL-GPU or CUDA

Postby ChristianZ » Wed Feb 22, 2017 7:30 pm

A bit of curiosity, how many particles were on the scene when the crash happened?


Sorry, I don't know exactly because I have deleted that project already, but it was more than 3 millions with that card with 4GB RAM.
RF10 standard, System specs: OSX 10.11, MacPro 4.1 2xXeon, GTX980Ti, 64 GB RAM


Return to “Realflow bug report”

Who is online

Users browsing this forum: No registered users and 1 guest