How to generate 3d assets from text and images with Flux and Trellis in NVIDIA Omniverse

Generative AI has come a long way, making it easier than ever to create high-quality 3D assets from text or image prompts. The combination of the Flux text-to-image pipeline by Blackforest Labs and the Trellis image-to-3D pipeline from Microsoft Research provides a seamless way to generate stunning, textured 3D models with just a few clicks in NVIDIA Omniverse.

As a platform designed for developing digital twins, NVIDIA Omniverse offers a perfect environment for integrating these pipelines. With KIT 106.5 Extensions, you can unlock the full potential of these tools to streamline your 3D asset creation workflow.

Text to Image Extension

text to image generation with Flux

To generate an image:

1. Enter your prompt.
2. Click the “Generate Image” button.

The Flux pipeline will be started locally, and the generated image will be saved in the default directory. If you want to specify a different output directory, use the folder selection button on the left.

If you’re running the text-to-image render service on a separate machine (using the genai.services.kit app, see details below), ensure the extension connects to it. Open the settings dialog via the button on the right and check the appropriate box to enable the remote service.

settings for the render service client

Image to 3D Extension

image to 3D generation with Trellis

Creating a 3D model from an image is straightforward:

1. Select an image using the button on the right.
2. Click “Generate 3D”.

The Trellis pipeline will process the image and load the resulting 3D model directly into the application for further use.

Note:

• The Flux pipeline requires 32GB of GPU RAM, while the Trellis pipeline needs 8GB.
• This setup has been successfully tested on two workstations equipped with NVIDIA RTX A6000 graphics cards. In our tests, one workstation ran the Text-to-Image Render KIT service, while the other powered the GenAI app.

Prerequisites

Here are a few prerequisites you need to install — please also note that the first start of the extensions can take very long because large models are downloaded into cache from hugging face. This may take up to several hours depending on your internet connection.

• download and install git

• download and install CUDA 12.4 for your platform from here:

GenAI Omniverse App

# check out app repository 
git clone https://github.com/perfectproducts/OmniGenAI.git

# build app and download dependencies 
cd OmniGenAI
# build app 
repo build
# run app and download additional dependencies, (-d adds developer extensions)
repo launch -d

Select “genai.toolkit.kit” to run the app — you can run a local flux render service with “genai.services.kit”.

alternativeliy you can run the app with the command

repo launch genai.toolkit.kit

GenAI Services

In order to use the flux pipeline set the environment variable HUGGINGFACE_HUB_TOKEN with your token before launching the services backend.

export HUGGINGFACE_HUB_TOKEN=[your token]
# launch the service backend 
repo launch genai.services.kit 

The service backend uses the listen port 8011. If you want to use a different port run it like that:

repo launch your_application.kit — — /exts/omni.services.transport.server.http/port=8888

By leveraging the power of NVIDIA Omniverse, Flux, and Trellis, you can quickly transform ideas into highly detailed 3D assets. Whether for prototyping, visualization, or creative projects, this workflow makes it easier than ever to turn text and images into stunning 3D creations.

Have fun exploring the possibilities!