Accessing Export Central Variables through a batch Script

Discussion about scripting with Python
lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Accessing Export Central Variables through a batch Script

Postby lukeiamyourfather » Wed Jun 17, 2009 6:42 pm

mdonovan wrote: [quote=lukeiamyourfather]The whole sequence of simulating on a node could be initiated in RealFlow with a GUI though, a batch script. However you do it try to keep projects self contained and not split up assets from the scene file. Cheers!

Sorry if this is obvious ... but how do I do the whole thing from a batch within realflow ... I mean how to I execute the remote command line from within my local realflow ?

man I fell like a noob.[/quote]
Python itself can directly interact with the command prompt or shell. The os module (os.system() which is kind of old school) or subprocess module (subprocess.Popen() replaced os.system() with more features like piping) can launch things, or submit tasks to a queue manager, whatever can be done directly in the command prompt or shell. Also some queue managers have Python API so you can submit tasks directly form Python without using the shell or command prompt.

Ideally simulation tasks should be sent through a queue manager just like renders would be, but if you have to send it to a single remote system explicitly then do it with ssh or something like that. Not sure if Windows has a remote shell equivalent but it probably does. Not to dog Windows too much but this is where using Linux in a production environment is just meant to be. Cheers!


mdonovan
Posts: 121
Joined: Wed Jun 17, 2009 3:59 pm

Accessing Export Central Variables through a batch Script

Postby mdonovan » Wed Jun 17, 2009 7:01 pm

ahhhhh ... this is starting to make sense ... so correct my thought process.

1) From within realflow pop a gui that asks the user which machine he would like to simulate on.
2) Using python, copy the entire realflow project from my local machine to that remote sim machine.
3) Using python send a batch command to our render manager (i think it can execute command lines.. we are currently using Smedge).

The batch file sent to the remote node from the manager would contain some like ...

cd
cd rf_projects%1
realflownode %2
xcopy remotepathparticles networkpathparticles /s
cd ..
deltree %1

I would much rather use just python than any dos commands, but i dont know how ... yet.

lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Accessing Export Central Variables through a batch Script

Postby lukeiamyourfather » Wed Jun 17, 2009 7:17 pm

Yeah, but maybe it would be best to copy the files to the network storage instead of from a workstation directly to a node. There's some room for error and people accessing the same files, but usually that's not much of a problem with only a few RealFlow artists on a show.

1. Artist downloads or creates new scene on local workstation, script or manually.
2. Tweak and work locally until a remote simulation is needed.
3. Artist synchronizes local RealFlow scenes with network storage, using a script or manually.
4. This step could go a few directions. Either open the scene from the network and submit to the queue manager from the RealFlow GUI or write a tool to submit outside of RealFlow.
5. Render node gets the job from the queue manager which runs a Python script to copy the scene locally from the network. Then it launches the simulation using the local files. At the end of the simulation the Python script running everything puts the files back where they were.
6. Artist can check out the results, and make those into an asset in the pipeline so the non-RealFlow users can access them.

Fill in the details and automate things where you can but that's how I've done it in the past. cheers!

TinuZ
Posts: 29
Joined: Tue Mar 06, 2007 1:22 pm
Contact:

Accessing Export Central Variables through a batch Script

Postby TinuZ » Thu Jun 25, 2009 1:45 pm

I'm wondering about the speed difference you guys are talking about, in simulating locally or on the network, I've never tested this myself (doh!) so I'm curious about it..

I mean I have a sim running for 2 days now... ahum.. so if locally this is 1 day that would be worthwile ofc...

cheers

lukeiamyourfather
Posts: 2880
Joined: Mon Oct 15, 2007 4:09 pm
Contact:

Accessing Export Central Variables through a batch Script

Postby lukeiamyourfather » Thu Jun 25, 2009 4:52 pm

TinuZ wrote: I'm wondering about the speed difference you guys are talking about, in simulating locally or on the network, I've never tested this myself (doh!) so I'm curious about it..

I mean I have a sim running for 2 days now... ahum.. so if locally this is 1 day that would be worthwile ofc...

cheers

The simulation itself spends most of the time calculating and not doing anything with the disk. So say it calculates for 20 hours and writes stuff to the network for two hours, that's 22 hours total. Working with a local RAID array you might get 20 hours of simulation and 30 minutes of writing to the disk, so 20.5 hours total saving 1.5 hours. That's just a rough example but you get the idea. It also matters for things like previewing a simulation or meshing.

If a simulation is 100GB it might take half a day to preview it over a typical gigabit network but if everything is on a local array it might take an hour or less. It just takes that long to transfer all of the gigabytes of data. Cheers!

mdonovan
Posts: 121
Joined: Wed Jun 17, 2009 3:59 pm

Accessing Export Central Variables through a batch Script

Postby mdonovan » Mon Jun 29, 2009 1:07 pm

I agree ... one of the major advantages to local->copy method is that you can see if your result is usable in a quicker timeframe. So if you are trying variations on multiple remote nodes u can see many results quicker and then copy the ones you like to the network for rendering. Also ... meshing local is faster too I believe.


Return to “Python-Scripting”

Who is online

Users browsing this forum: No registered users and 2 guests