Multithreaded meshing process updated

Share your scripts with other members
lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Multithreaded meshing process updated

Postby lukeiamyourfather » Tue Feb 10, 2009 1:26 am

I've converted the script to Python 3.0 (not reverse compatible with previous versions). It now has support for task size or increment (range of frames) which greatly improves performance. There's a bug in RealFlow where it'll mesh the first frame given in the command line two times no matter what, so the increment reduces the "lost" performance due to this. Only for Linux right now. Anybody want to try it out? The script is below with directions included. I'd like to get a second opinion before posting it on the Next Limit scripting site. Cheers!

Code: Select all

"""
multiThreadedMeshProcess.py

Creating meshes in RF4 is a single threaded process out of the box. To speed meshing this script will launch multiple threads of RF4 in command line mode. Each thread will process a range of frames until all frames have been meshed. Starting with the first frame parameter given, and meshing the next unmeshed frame range until the last frame parameter given. All frames in the range given including the first and last frame will be meshed.

This is an improvement on previous meshing scripts available because it doesn't write any extra files when executing the script, is not platform specific (relatively speaking), and meshes one frame at a time instead of a range of frames. If meshes are denser towards the end of a simulation, the work would not be distributed evenly with previous scripts which means wasted potential processing resources. With meshing one frame at a time or a small frame range until all are completed, the work is more evenly distributed to available processors.

This script was developed with RealFlow 4.3.8.0124 64-bit and Python 3.0 on CentOS 5.2 64-bit. Earlier versions of Python will not work. The path to RF4 might have to be modified for the script to function, or add the RealFlow directory to the system path (usually located in ~/.bash_profile). Testing has not been performed on other platforms but it should work on other Linux systems. The realflow executable can be changed to realflownode for command line licenses of RF, also remove the -nogui flag.

If using a Linux distribution that doesn't have Python 3.0 already, download the source from Python.org and use make altinstall to install Python 3.0 without overwriting the currently installed version of Python. If the script runs very quickly and no meshes are generated, then the RealFlow path or the scene path are probably incorrect.

An example terminal command is given below:

python3.0 *pathToScript*/multiThreadedMeshProcess.py *pathToScene*/testScene.flw *firstFrame* *lastFrame* *increment **threadCount* *niceness*
python3.0 ~/multiThreadedMeshProcess.py /scenes/meshingTest/meshingTest.flw 0 200 5 8 10

Thanks to Brad Dayley, his book Python Phrasebook came in very handy. Also thanks to realflowforum.com users for their generous Python help.

Author: Luke Olson (luke.s.olson@gmail.com or lukeo@cafefx.com)
Date: 2009-02-09
"""


import sys
import queue
import threading
import time
import _thread
import subprocess

#Switch to 1 when all frames have completed
doExit = 0

#Dump subprocess output to /dev/null to keep the terminal window output clean
FNULL = open("/dev/null", "w")

#Thread class to launch threads
class newThread (threading.Thread):
   def __init__(self, threadID, name, q, scene, niceValue):
      threading.Thread.__init__(self)
      self.threadID = threadID
      self.name = name
      self.q = q
      self.scene = scene
      self.niceValue = niceValue
   def run(self):
      queueLock.acquire()
      print(timeStamp() + " " + str(self.name) + " Launched")
      queueLock.release()
      processMesh(self.name, self.q, self.niceValue, self.scene)
      queueLock.acquire()
      print(timeStamp() + " " + str(self.name) + " Closed")
      queueLock.release()

#Function to mesh frames, called in each new thread
def processMesh(tName, q, niceValue, scene):
   while not doExit:
      queueLock.acquire()
      if not workQueue.empty():
         data = q.get()
         print(timeStamp() + " " + str(tName) + " Started meshing frame " + str(data[0]) + " - " + str(data[1]))
         queueLock.release()
         #Launch RealFlow and mesh frame, wait until complete
         subprocess.Popen("nice -n %s realflow -nogui -threads 1 -range %s %s -mesh %s" % (niceValue, data[0], data[1], scene), stdout=FNULL, stderr=FNULL, shell=True).wait()
         queueLock.acquire()
         print(timeStamp() + " " + str(tName) + " Finished meshing frame " + str(data[0]) + " - " + str(data[1]))
         queueLock.release()
      else:
         queueLock.release()

#Function to return the current time
def timeStamp():
   return str(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(time.time())))

#Gather input from the command line arguments
sceneFile = sys.argv[1]
firstFrame = sys.argv[2]
lastFrame = sys.argv[3]
increment = sys.argv[4]
threadCount = sys.argv[5]
niceness = sys.argv[6]

#Lists of threads and frames for meshing
threadList = []
frameList = []

#Populate frameList
currentFrame = int(firstFrame)
while currentFrame <= int(lastFrame):
   firstTask = currentFrame
   lastTask = currentFrame + int(increment) - 1
   if lastTask > int(lastFrame):
      lastTask = int(lastFrame)
   frameList.append([firstTask, lastTask])
   currentFrame += int(increment)

#Populate threadList
for thread in range(0, int(threadCount)):
   threadList.append("Thread-" + str(thread))

#Multi threading locking and queue
queueLock = threading.Lock()
workQueue = queue.Queue(0)
threads = []
tID = 1

#Create new threads
for tName in threadList:
   thread = newThread(tID, tName, workQueue, sceneFile, niceness)
   thread.start()
   threads.append(thread)
   tID += 1

#Fill the queue
queueLock.acquire()
for frame in frameList:
   workQueue.put(frame)
queueLock.release()

#Wait for queue to empty checking the status once every .01 seconds
while not workQueue.empty():
   time.sleep(.01)

#There is no more work in the queue, ready to exit
doExit = 1

#Close all threads
for t in threads:
   t.join()

print(timeStamp() + " Meshing complete and now exiting")

#End of script
sys.exit(0)


Attached files


ranxerox
Posts: 334
Joined: Tue Jan 09, 2007 5:15 pm

Multithreaded meshing process updated

Postby ranxerox » Tue Feb 10, 2009 5:24 pm

hey Luke, that script looks exciting ! I must admit I haven't used python 3.0 yet (stuck on older version due to various application restrictions etc.) Looks very nice indeed. I would like to get into multithreading in python myself, haven't written any threaded code in a while. One thing about usage which jumps out at me, it would be good to be able to supply the actual command which runs realflow on the command line (as an option). This might be in case the user wants to run different versions (beta or older versions) or something like realflownode instead of realflow -nogui (I usually use the nodes for meshing for example).

looks good, I'll have to try to upgrade to python 3.0 to try it though, write now I use a shell script to do multithreaded meshing.

thanks

-ranxx

lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Multithreaded meshing process updated

Postby lukeiamyourfather » Tue Feb 10, 2009 5:36 pm

Be sure not to overwrite the original Python on the system :ninja:

Otherwise all kinds of things break. Downloading the source for Python 3.0 is the easier way to upgrade. Drop the source into /usr/local/src and then run this in the Python 3.0 source directory:

Code: Select all

./configure
make altinstall

After that you'll get python3.0 in the terminal that won't overwrite any other Python versions. Using just python will go to the original version on the system.

There's really not that much different about it, a few modules have changed names and the print is actually a function now. The changes are deeper than that, but are easy to pick up. Cheers!

overload
Posts: 176
Joined: Mon Dec 18, 2006 5:00 pm

Multithreaded meshing process updated

Postby overload » Tue Feb 10, 2009 6:42 pm

Wanted to try it this morning, but still waiting for them to put python3.0 on my machines. But the previous version kicks ass, so I'm cool with that for now! Thanks Luke!!

shaun_michael
Posts: 4766
Joined: Sun Sep 10, 2006 8:04 am

Multithreaded meshing process updated

Postby shaun_michael » Tue Feb 10, 2009 9:29 pm

But the previous version kicks ass

Ouch! :ninja:

Good to see you again Anthony.

Shaun

lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Multithreaded meshing process updated

Postby lukeiamyourfather » Thu Feb 12, 2009 11:30 pm

I've posted this script at the scripting site.

http://www.nextlimit.com/nlscript/news_scripts.php?date=12/02/2009&id=127

And also a command line simulation launcher for GNOME systems (previous version used KDE).

http://www.nextlimit.com/nlscript/news_scripts.php?date=12/02/2009&id=128

Cheers!

lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Multithreaded meshing process updated

Postby lukeiamyourfather » Fri Feb 13, 2009 7:07 pm

Here's a version that works on Windows if somebody wants to try it out. Will post it soon on the scripting site once it gets tested. All it needs is Python 3.0 on the system and RealFlow installed.

Python 3.0 can be downloaded from:

http://www.python.org/download/

Directions are in the head of the script. Cheers!

Code: Select all

"""
multiThreadedMeshProcess2Windows.py

Creating meshes in RF4 is a single threaded process out of the box. To speed meshing this script will launch multiple threads of RF4 in command line mode. Each thread will process a range of frames until all frames have been meshed. Starting with the first frame parameter given, and meshing the next unmeshed frame range until the last frame parameter given. All frames in the range given including the first and last frame will be meshed.

This is an improvement on previous meshing scripts available because it doesn't write any extra files when executing the script, is not platform specific (relatively speaking), and meshes one frame at a time instead of a range of frames. If meshes are denser towards the end of a simulation, the work would not be distributed evenly with previous scripts which means wasted potential processing resources. With meshing one frame at a time or a small frame range until all are completed, the work is more evenly distributed to available processors.

This script was developed with RealFlow 4.3.8.0124 64-bit and Python 3.0 on Windows XP Professional x64 Edition. Earlier versions of Python will not work. The path to RF4 might have to be modified for the script to function depending on the install location. Testing has not been performed on other platforms but it should work on other Windows systems. The realflow executable can be changed to realflownode for command line licenses of RF, also remove the -nogui flag.

If using a system that doesn't have Python 3.0 already, download the installer from Python.org.

An example terminal command is given below:

C:Python30python.exe "*pathToScript*multiThreadedMeshProcess2Windows.py" "*pathToScene*testScene.flw" *firstFrame* *lastFrame* *increment *threadCount*
C:Python30python.exe "C:scriptsmultiThreadedMeshProcess2Windows.py "C:scenesmeshingTest.flw" 0 200 5 8

Thanks to Brad Dayley, his book Python Phrasebook came in very handy. Also thanks to realflowforum.com users for their generous Python help.

Author: Luke Olson (luke.s.olson@gmail.com or lukeo@cafefx.com)
Date: 2009-02-13
"""


import sys
import queue
import threading
import time
import _thread
import subprocess

#Switch to 1 when all frames have completed
doExit = 0

#Dump subprocess output to /dev/null to keep the terminal window output clean
FNULL = open("NUL")

#Thread class to launch threads
class newThread (threading.Thread):
   def __init__(self, threadID, name, q, scene):
      threading.Thread.__init__(self)
      self.threadID = threadID
      self.name = name
      self.q = q
      self.scene = scene
   def run(self):
      queueLock.acquire()
      print(timeStamp() + " " + str(self.name) + " Launched")
      queueLock.release()
      processMesh(self.name, self.q, self.scene)
      queueLock.acquire()
      print(timeStamp() + " " + str(self.name) + " Closed")
      queueLock.release()

#Function to mesh frames, called in each new thread
def processMesh(tName, q, scene):
   while not doExit:
      queueLock.acquire()
      if not workQueue.empty():
         data = q.get()
         print(timeStamp() + " " + str(tName) + " Started meshing frame " + str(data[0]) + " - " + str(data[1]))
         queueLock.release()
         #Launch RealFlow and mesh frame, wait until complete
         subprocess.Popen('realflow.exe -nogui -threads 1 -mesh -range %s %s "%s"' % (data[0], data[1], scene), stdout=FNULL, stderr=FNULL, shell=True, cwd='C:Program Files (x86)Next Limitx64RealFlow4').wait()
         queueLock.acquire()
         print(timeStamp() + " " + str(tName) + " Finished meshing frame " + str(data[0]) + " - " + str(data[1]))
         queueLock.release()
      else:
         queueLock.release()

#Function to return the current time
def timeStamp():
   return str(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(time.time())))

#Gather input from the command line arguments
sceneFile = sys.argv[1]
firstFrame = sys.argv[2]
lastFrame = sys.argv[3]
increment = sys.argv[4]
threadCount = sys.argv[5]

#Lists of threads and frames for meshing
threadList = []
frameList = []

#Populate frameList
currentFrame = int(firstFrame)
while currentFrame <= int(lastFrame):
   firstTask = currentFrame
   lastTask = currentFrame + int(increment) - 1
   if lastTask > int(lastFrame):
      lastTask = int(lastFrame)
   frameList.append([firstTask, lastTask])
   currentFrame += int(increment)

#Populate threadList
for thread in range(0, int(threadCount)):
   threadList.append("Thread-" + str(thread))

#Multi threading locking and queue
queueLock = threading.Lock()
workQueue = queue.Queue(0)
threads = []
tID = 1

#Create new threads
for tName in threadList:
   thread = newThread(tID, tName, workQueue, sceneFile)
   thread.start()
   threads.append(thread)
   tID += 1

#Fill the queue
queueLock.acquire()
for frame in frameList:
   workQueue.put(frame)
queueLock.release()

#Wait for queue to empty checking the status once every .01 seconds
while not workQueue.empty():
   time.sleep(.01)

#There is no more work in the queue, ready to exit
doExit = 1

#Close all threads
for t in threads:
   t.join()

print(timeStamp() + " Meshing complete and now exiting")

#End of script
sys.exit(0)

lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Multithreaded meshing process updated

Postby lukeiamyourfather » Sat Feb 14, 2009 8:15 am

A version that runs on Windows has been added to the Next Limit scripting site, and there's a video showing how each works and how to install Python 3.0 on Linux. My blog post about it with the video download:

http://whenpicsfly.com/wordpress/?p=81

Sorry to keep dragging this thread out but I think its good for a while now. Cheers!

BROTSPINNE
Posts: 1
Joined: Wed Jun 10, 2009 3:48 pm

Multithreaded meshing process updated

Postby BROTSPINNE » Tue Jun 23, 2009 11:43 am

That is really good script to speed things up. But the script is just working to frame 100. Until that the mesh is prefect. After this frame the bin file is 1kb.

Please can somebody help me?

lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Multithreaded meshing process updated

Postby lukeiamyourfather » Tue Jun 23, 2009 4:48 pm

BROTSPINNE wrote: That is really good script to speed things up. But the script is just working to frame 100. Until that the mesh is prefect. After this frame the bin file is 1kb.

Please can somebody help me?

Are there particles after frame 100? If the mesh files are being generated (even if they are empty) then the script is doing its job. Double check the particles and scene settings (frame ranges, mesh settings, etc.). Cheers!


Return to “User Scripts”

Who is online

Users browsing this forum: No registered users and 1 guest