teacup's techy Art stuff

My Zbrush to Maya Workflow

I’ve been using Zbrush for a few years now, usually just for small chunks here and there, and I’ve never found a workflow that I was happy with. There always seemed to be some compromise in the methods; whether it was just the amount of work required to do the export, different features missing, or discarding some of the detail of the Zbrush sculpt.

But, I’ve finally found a worflow that seems to work well; low and high poly meshes can be modelled completely seperately, baking is done in Maya which has the best cage/projection method in my opinion, all the details from the sculpt in Zbrush are retained, and it’s fairly straightforward and quick.

The only downside to this method is that it doesn’t give you a totally precise result, because we’re using a displacement map, and manipulating and creating the normal map in Photoshop, the details aren’t transfered as accurately as if they were baked directly from high to low meshes. I can’t really notice any problem, and for the environment work I do it’s easily close enough, but I guess for a highly detail face then slight differences may be a problem.

Summary/checklist

• ZBrush
• Create high res in Zbrush
• Switch to lowest subdiv
• AUVTiles
• Export Low poly
• Switch to medium subdiv
• Export medium poly
• CreateExport Displacement map
• Switch to highest subdiv
• CreateExport color texture
• Photoshop
• Flip color vertically and resave
• Flip displacement vertically, Auto-levels, convert to 8bit channels, resave
• Maya
• Import/prepare low poly mesh (UVs)
• Import medium poly
• Transfer maps to create normal map
• Assign displacement map to color on medium poly, transfer maps to get new displacement
• Assign poly paint texture to color on medium poly, transfer maps to get new color
• Transfer maps to create Ambient occlusion
• Photoshop
• Use new displacement with nvidia plugin to create new normal map. Overlay this with the first one
• Use new displacement to add detail to the ambient occlusion

In-depth walk through

In ZBrush

Create super high res model in Zbrush, poly paint the texture if required. Don’t worry about UVs, topology, etc. Doesn’t really matter how you start off this model, whether from a cube or ZShperes in Zbrush, or a mesh made in Maya, we’re only really interested in the high poly mesh.

Switch to the lowest subdiv.

Create UVs for the model. If you already have UVs on the model you can skip this step, but be aware that if the UVs are badly laid out then you may lose detail from the highest subdivs. By creating new UVs now you are making sure they’re good quality.

Tool -> Texture AUVTiles (may have to Enable UVs on this panel if unavailable) If the poly sizes vary a lot its worth increasing the AUVRatio settings, this will keep the amount of texture space more even.

Export Low Poly mesh or whichever subdiv you want to use for your final low poly mesh. This is to use as you final low poly mesh, you can export the lowest and use it directly, or use a higher subdiv as a guide to modeling the mesh (e.g. with Topogun, or Nex), or just skip this if you’ve already got a mesh you want to use.

Switch to a medium level subdiv, this will be the mesh that is imported into Maya so you want something that it can handle well, but doesn’t smooth out too much detail. I’ve found Maya is fine up to around a million points, after that it gets a bit sluggish, and around two million it starts getting unstable. Two levels below the highest subdiv level seems to work well.

Export this as your high poly mesh

Create the detail displacement map (Tools -> Displacement -> Create DispMap)  This will contain the differences between the medium poly mesh you exported and the highest subdiv. It’s best to use a texture size double that of your final required size to make sure no details are lost when transferring to the low poly mesh (DPRes setting). You can experiment with the Adaptive and SmoothUV settings, their description sound like they should improve the result but I couldn’t really see any difference 🙂

After clicking ‘Create DispMap’ the texture will be added as an alpha texture. Select this and export it. The default export will create a 16bit PSD file which seems to work fine.

Switch to highest subdiv level to export the poly paint as a texture, you can skip this if you’ve not poly painted your model. First create a new texture, as with the displacement map I think it’s worth setting this to twice the size of your final size. (Texture->Width/Height->New) Then copy your poly paint to this texture (Tools->Texture->Col>Txr) and then export it.

We’re finished in Zbrush now, so save your tool and quit.

In Photoshop

Zbrush outputs textures upside down so we need to flip them the right way up. (Image->Rotate Canvas->Flip Canvas Vertical) Flip the color texture and re-save.

For the displacement map we need to do a little work, but first it still needs to be flipped. Then do an Auto-Levels (Image->Adjustments->Auto-Levels) this will  ensure the useful information is using the full range of our texture. If the texture is still very grey then you can bump it up further with the levels tool, but be aware that you will then start losing information in the high and low parts of the heightmap.

Convert the texture to 8 Bits/Channel (Image->Mode->8 Bits/Channel), this will change the texture to a normal color texture that we can use with the Nvidia Normal map Plugin later. Save. And we’re done with Photoshop for now.

In Maya

You need to get your final low poly mesh now, where it comes from doesn’t really matter (the exported mesh from Zbrush, or hand crafted from the high poly) It needs to have the final UV layout that is suitable for baking. We will be transferring the details and textures we’ve already created to this new low poly mesh.

Import the high poly mesh you exported from Zbrush. You need to make sure the two meshes are correctly aligned, if they were both exported from Zbrush they should be correct already. Once they are aligned do a freeze transforms (Modify->Freeze Transforms) making sure to freeze translate, rotate, and scale. Once you’ve done this you can move the meshes around separately and Transfer Maps will still work correctly which can be handy to compare the results.

We’re going to use Mayas Transfer Maps tool (Lighting/Shading->Transfer Maps) to bake all of our nice fine details onto the final UV layout on our low poly model. (Pre Maya 8 this tool was called Surface Sampler, it has the same features but things might be called slightly differently)

Set you low poly mesh as the target mesh, select the correct UV set if necessary, and add the high poly as the source mesh. Click the ‘Normal’ button to set up the transfer for a normal map, you probably want to make sure the ‘Map space’ is set to ‘Tangent space’.

You can set the resolution, and other details in the ‘Maya Common Output’ section. Then hit the Bake button and it will spit out a nice medium detail normal map.

Now we’re going to use Transfer maps to convert our displacement and poly paint textures so that they match up with the UV set on the low poly mesh. Assign the displacement map to the color channel on the high poly mesh shader, then use the ‘Diffuse’ mode in Transfer maps to bake it out. Then do the same for the polyPaint texture if you need to.

You can also get an Ambient occlusion texture baked out using the ‘Ambient’ button. See this post for a hack that adds the ‘Spread’ attribute to the options.

In Photoshop

You should now have a selection of textures that match your low poly model, that contain all the details from the high poly sculpt. The last thing to do is to get the details from the displacement map on to the normal map.

Use the displacement map to create a new normal map with the nvidia normal map plugin. You’ll have to play with the intensity till you get a value that looks good, because of the way we made this displacement map there is no ‘right’ value, it will be different for each model.

Then overlay this normal map onto the one you baked from Maya. You can do a simple overlay by filling the blue channel with 50% grey set to multiply, then just set the layer to Overlay. Or you could do a more complex ‘correct’ overlay, this is explained here, where you can also just download an action to do it.

And one last little tip: the detail height map can give a nice extra bit of detail to the ambient occlusion, just need to play around with the blends a little.

Hopefully that all made sense, and you can get some nice results from it!

Maya Paths & Customization

I’ve spent a lot of time figuring out how to customize Maya; it really has an excellent system allowing you to change almost anything. The underlying system for allowing this is based on several Path variables that Maya will check to see where it should load its functionality from. With clever use of these it’s possible to create a predictable, modular system for allowing customization from different projects, or for individual users. I’ve documented how Maya handles each of the important Path variables, and have a couple of suggestions for how to set them up.

Diagnosing Path Issues

From within Maya you can use the mel commands ‘getenv’ and ‘putenv’ to get and store environment variables. Changing the variables like this is volatile, it’s only for this instance of Maya. You can use the ‘whatIs’ command to query where a specific script or command is being run from.

I’ve also written a little script to give a nice UI for checking the environment variables – envBrowser

Variables

MAYA_APP_DIR  – default is `%USERPROFILE%/My Documents/maya` MAYA_LOCATION – Defaults to the directorys that Maya is being run from (eg, `%PROGRAMFILES%/Autodesk/maya<version>`) This can be changed, but must point to a directory containing a valid Maya install, if not Maya will revert back to defaults.

Script Paths – MAYA_SCRIPT_PATH

When trying to find a script or command Maya will search through the script path from top to bottom until it finds it, then stop. This means that scripts higher up the path will override lower ones, including any of Mayas own default scripts. So to edit an installed script you can copy it to your local My Documents script folder and it will then ignore the original.

• Current Project
  • The default if you haven’t set your own is: %MAYA_APP_DIR%/projects/default. Annoyingly if you set a mel directory in the project settings then it doesn’t add it to the scripts path, and Maya won’t find your scripts.
• User environment variables System environment variables
  • It will only look at the first one of these, then skip the rest.
• Maya.env %MAYA_APP_DIR%/<version >/Maya.env %MAYA_APP_DIR%/Maya.env
  • It will only look at the first one of these, then skip the rest. Can only define one MAYA_SCRIPT_PATH and it will use the first it comes to within the file, any others will be ignored. If you want to add multiple directories then add them all to one line, using semi-colons to seperate them.
• %MAYA_APP_DIR%/<version>/scripts
• %MAYA_APP_DIR%/scripts
• %MAYA_APP_DIR%/<version>/presets
• Shelf paths
  • All locations from %MAYA_SHELF_PATH% are added here
• %MAYA_APP_DIR%/<version>/prefs/shelves
• %MAYA_APP_DIR%/<version>/prefs/markingMenus
• %MAYA_LOCATION%/scripts/startup
• %MAYA_LOCATION%/scripts/others
• %MAYA_LOCATION%/scripts/AETemplates
• %MAYA_LOCATION%/scripts/paintEffects
• %MAYA_LOCATION%/scripts/fluidEffects
• %MAYA_LOCATION%/scripts/hair
• %MAYA_LOCATION%/scripts/cloth
• %MAYA_LOCATION%/scripts/fur
• Modules

userSetup.mel Maya will only run this script once, and will use the first one it comes to in the path system. To allow us to define a team userSetup but still allow a user to run their own scripts, we use a userSetup file that is provided by the team project. After completing team setup, this file will try to run ‘localSetup.mel’ which the user can customize.

Module Paths – MAYA_MODULE_PATH

Maya searches ‘MAYA_MODULE_PATH’ for module definition files; a text file that contains the name of the module and the location it is installed. For each module path found, it appends the suffixes “plug-ins”, “presets”, “scripts”, and “icons”, and adds them to MAYA_PLUG_IN_PATH, MAYA_PRESET_PATH, MAYA_SCRIPT_PATH (and PYTHONPATH), and XBMLANGPATH, respectively.

• User environment variables System environment variables
  • It will only look at the first one of these, then skip the rest.
• %MAYA_APP_DIR%/<version>/modules
• %MAYA_APP_DIR%/modules
• Maya’s doing something weird for this next set of paths, it seems it might be defined in the registry, and a newer version of maya will continue to use an older directory if it still exists. So this could be in different places on different machines.
  • %PROGRAMFILES%/Common Files/Alias Shared/Modules/maya/<version>
  • %PROGRAMFILES%/Common Files/Alias Shared/Modules/maya OR
  • %PROGRAMFILES%/Common Files/Autodesk Shared/Modules/maya/<version>
  • %PROGRAMFILES%/Common Files/Autodesk Shared/Modules/maya
• %MAYA_LOCATION%/modules

Plug in Path – MAYA_PLUG_IN_PATH

Any directories listed here will be added to the Plugin Manager window in Maya, however, in the same manner as scripts, Maya will use the first plugin it finds in the list, and then ignore any further plugins named the same. Even when selecting to autoload a specific plugin, when restarting Maya the ‘autoload’ checkbox will move to the first plugin it finds. The only exception to this is when you load the plugin manually by clicking the ‘Loaded’ checkbox, it will correctly load the specified plugin. Also you can load a specific plugin in a script by stating the full path of the plugin to load.

• User environment variables System environment variables
  • It will only look at the first one of these, then skip the rest.
• %MAYA_APP_DIR%/<version>/plug-ins
• %MAYA_APP_DIR%/plug-ins
• %MAYA_LOCATION%/plug-ins
• Modules

Shelf Path – MAYA_SHELF_PATH

If you query this variable it doesn’t report Mayas default shelf directory, I think it only checks here: %MAYA_APP_DIR%/<version>/prefs/shelves

• User environment variables System environment variables Maya.env
  • It will only look at the first one of these, then skip the rest.
• Mayas own stuff

Icons – XBMLANGPATH

Maya searches the ‘XBMLANGPATH’ variable for icons, the variables ‘MAYA_ICON_PATH’ and ‘MAYA_FILE_ICON_PATH` are obsolete and are no longer used.

• User environment variables System environment variables
  • It will only look at the first one of these, then skip the rest.
• %MAYA_APP_DIR%/<version>/prefs/icons
• %MAYA_APP_DIR%/prefs/icons
• %MAYA_LOCATION%/icons
• %MAYA_LOCATION%/app-defaults
• %MAYA_LOCATION%/icons/paintEffects
• %MAYA_LOCATION%/icons/fluidEffects
• %MAYA_LOCATION%/icons/hair
• %MAYA_LOCATION%/icons/cloth
• %MAYA_LOCATION%/icons/live
• %MAYA_LOCATION%/icons/fur
• Modules

Python scripts – PYTHONPATH

Similar to mel scripts, Maya will search through the path until it finds what it’s looking for, then will stop. This allows you to override scripts by placing them higher up the path. Note that ‘Modules’ is above the My Docs scripts which is different than for all the other paths. Mayas default locations aren’t shown in the PYTHONPATH variable, to see the full path that python is using, query the sys.path variable in Python.

• %MAYA_LOCATION%/bin
• User environment variables System environment variables
  • It will only look at the first one of these, then skip the rest.
• Modules
• %MAYA_LOCATION%/bin/python25.zip
• %MAYA_LOCATION%/Python/DLLs
• %MAYA_LOCATION%/Python/lib
• %MAYA_LOCATION%/Python/lib/plat-win
• %MAYA_LOCATION%/Python/lib/lib-tk
• %MAYA_LOCATION%//bin
• %MAYA_LOCATION%/Python
• %MAYA_LOCATION%/Python/lib/site-packages
• %MAYA_APP_DIR%/<version>/prefs/scripts
• %MAYA_APP_DIR%/<version>/scripts
• %MAYA_APP_DIR%/scripts

userSetup.py This is handled slightly differently than the mel equivalent, Maya will execute all userSetup.py files it finds in the PYTHONPATH directories, they will be ran in the order found. Also, this file is ran before Mayas UI is created, so cannot be used to access or hack any of the default Maya UI. Any print commands will be output to the ‘Output Window’ instead of the Script Editor History.

Breaking infinite loops in MEL & a better ScriptEditor

I found this post linked to on the CGTalk forums, about a way to recover from an infinite loop in MEL; using a filetest inside the loop see if a ‘kill’ file existed, if it did then the loop would end. It got me thinking about other ways to help mitigiate this problem, and also about how I don’t really like the script editor as it is (Why does it delete the script if it’s not selected when I execute?).

This article uses commands that were added in Maya 2008, if you’re still using an earlier version it should still be possible to use a similar technique but more scripts would need to be written to fill in the gaps.

Problem 1. Backing up the ScriptEditor contents

The first problem is for when you’ve forgotten to put in a check for loops, or used the above method to get out of it, we obviously don’t want to loose the script we are working on when Maya has hung. Maya already saves the contents of the script editor when it shuts down, and the commands that accomplish this are in “scripts\startup\scriptEditorPanel.mel”, specifically the command “syncExecuterBackupFiles()”. So we could just add this command to the top of our script, and everytime we run it the editor is backed up.

But this could cause us some problems when we start Maya up again; this command saves out a file for each tab in the editor, however if we added or removed any tabs then this wouldn’t be saved when we killed Maya. When we opened up Maya again it would try and copy the saved contents back into the tabs, and if they had changed it might put mel files in a python tab, or just miss out some altogether. We wouldn’t loose anything, the contents would still be in the preferences folder, but it could be a pain to try and sort them out again so it’s not ideal. We could just run “SavePreferences” to save the layout of the tabs, unfortunately this takes a few seconds and spams the script history so it’s not a great solution either.

There are a couple of different approaches now, the simplest would be to create our own version of “syncExecuterBackupFiles()” that saved the contents out to our own directory, then if there was a crash you just check this folder and retreive the scripts manually.

Another option is to continue to use “SavePreferences” but instead of calling this everytime we execute a script, we call it whenever we alter the tabs themselves. There are three commands that create or remove tabs, (addNewExecuterTab, removeOtherExecuterTabs, removeCurrentExecuterTab) we just need to add “SavePreferences” as the last line in each. This is probably the ‘safe’ way to do it, all we’re doing is calling Mayas own back up commands, and it’s how it’s set up for me. But I still don’t really like it; it seems messy having to save all preferences, when all you’ve done is added a new tab in the editor.

My other idea involves altering the backup command so that as well as saving the contents of the editor, it also saved out the tab information. This would also require altering the code that builds the tabs so that it used this file instead of the normal user preferences. I think this would be a more elegant solution, the end user wouldn’t notice any difference and nothing would get printed to the script output. I might look into this further if I get a chance, for now the previous solution will be fine.

Problem 2. Better Execute Command

The other big issue I have is with the command to actually execute the script; I normally press “Ctrl+a” then “Ctrl+Enter” to select all and execute. I just want a single button to press that executes without removing the script, and it would be nice if we didn’t have to add the command to backup our script editor contents manually, it should just do it when you execute, maybe it would be useful to save the scene automatically before too.

I thought this should be fairly easy, but now all the problems start 🙁

The first issue that I didn’t anticipate was that the command to store the contents of the script editor also selects all the text, loosing our previous selection. This means we can’t do a backup when we just want to execute a selection of our script. We could do a check to see if there is a selection, and then not do the backup in that case; but I don’t really like the inconsistency, sometimes you get a backup, sometimes not. The only way I could come up with to solve this would be to create our own backup command, bypassing the store command built into the editor. I’ll return to this in another post I think.

Second issue is about hotkeys, or the lack of them when in the ScriptEditor. As far as I can tell the hotkeys that control the ScriptEditor are seperate from the normal hotkeys, and are hardcoded somewhere I can’t access. This means I can’t change Ctrl+Enter to run our command, and I can’t create new hotkeys either. Bah.

So what can we change?

The execute command can be altered a little so that if there is no selection it will do an executeAll instead. This stops it removing the script if nothing was selected. Unfortunately editing this will only affect the execute button, and the command in the menu, Ctrl+Enter bypasses this.

We can add a ‘Backup Scripts’ button which we can hit to save the contents of the editor. This is just a button with the ‘syncExecuterBackupFiles’ that I mentioned earlier. I also added a ‘SavePreferences’ command, and ‘SaveScene’ to the button, so I can uses it as a ‘I’m about to do something dangerous’ button.

And there is the inconsistent backup and execute button that was already mentioned.

So I managed to add a couple of useful features, even if it’s not the pure awesomeness I had hoped for. 🙂 I’ll go away and have bit more of a think about it.


Download: scriptEditorPanel

Maya Transfer Maps Ambient Occlusion Hack

For some unknown reason Autodesk decided that when using the Transfer Maps tool to create an ambient occlusion texture, the ‘Spread’ attribute was unnecessary (even though it’s the main control for altering the intensity of the occlusion!).
So I’ve hacked it in, now when creating an ambient occlusion texture you get the attributes: Rays, Max Depth, and Spread.

I’ve made scripts for each version of Maya, make sure you use the right one.

Download: performSurfaceSampling

Hello world!

Blurb about who I am.