Hermes/comfyui-trellis2-rocky-setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial: ComfyUI + Trellis2 on Rocky 9.x / RTX 4090

Browse Source 
Documents all pitfalls encountered during setup:
- CUDA arch mismatch (sm_120 vs sm_89) — source rebuild needed
- utils3d empty pip package → GitHub fork required
- transformers v5 DINOv3 API change → sed patch
- xformers version pinning (0.0.33.post2 for torch 2.9.1)
- Missing Eigen headers for cumesh build
- visualbruno forks needed for o_voxel and cumesh (newer functions)
- /tmp disk space issues during pip builds

Includes:
- SETUP.md: Full step-by-step guide
- requirements.txt: Pinned dependency versions
- fix_transformers_v5.sh: Apply DINOv3 layer patch
- build_cuda_extensions.sh: One-shot source build of all CUDA extensions
This commit is contained in:
Hermes
2026-06-07 22:27:26 +00:00
parent 58da752e26
commit c4057c261e
5 changed files with 500 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+47 -2
View File
@@ -1,3 +1,48 @@
# comfyui-trellis2-rocky-setup # ComfyUI + Trellis2 on Rocky Linux 9.x / RTX 4090
ComfyUI + Trellis2 on Rocky Linux 9.x / RTX 4090 — all pitfalls, fixes, working config **Working setup as of June 2026.** Documents every pitfall, fix, and the exact working configuration
for running [ComfyUI-Trellis2](https://github.com/visualbruno/ComfyUI-Trellis2) on Rocky Linux 9.7
with an NVIDIA RTX 4090 (sm_89 / Ada Lovelace).
## Quick Reference
| Component | Source | Build | Version |
|-----------|--------|-------|---------|
| CUDA Toolkit | DNF (`cuda-toolkit-12-8`) | system | 12.8 |
| PyTorch | pip `--index-url https://download.pytorch.org/whl/cu128` | prebuilt | 2.9.1+cu128 |
| flex_gemm | `JeffreyXiang/FlexGEMM` (GitHub) | **source (sm_89)** | 1.0.0 |
| o_voxel | `visualbruno/TRELLIS.2#subdirectory=o-voxel` | **source (sm_89)** | 0.0.1 |
| cumesh | `visualbruno/CuMesh` (GitHub) | **source (sm_89)** | 1.0 |
| nvdiffrast | `NVlabs/nvdiffrast` (GitHub) | **source (sm_89)** | 0.4.0 |
| xformers | pip `--index-url https://download.pytorch.org/whl/cu128` | prebuilt | 0.0.33.post2 |
| natten | `-f https://whl.natten.org` | prebuilt | 0.21.5+torch290cu128 |
| utils3d | `EasternJournalist/utils3d.git@3fab839` | source | 1.3 |
## Directory Structure
```
~/Documents/ComfyUI_trellis2_pixal/
├── venv/ # Python 3.12 virtual environment
├── main.py # ComfyUI entry point
├── custom_nodes/
│ └── ComfyUI-Trellis2/ # visualbruno/ComfyUI-Trellis2
├── models/
│ ├── facebook/
│ │ └── dinov3-vitl16-pretrain-lvd1689m/ # DINOv3 model (download manually)
│ └── trellis2/
│ └── TRELLIS.2-4B/ # TRELLIS.2-4B model (download manually)
└── conda-libs/ # Conda libstdc++ workaround (legacy)
```
## Start ComfyUI
```bash
cd ~/Documents/ComfyUI_trellis2_pixal
source venv/bin/activate
python main.py --listen 127.0.0.1
# GUI at http://127.0.0.1:8188
```
## Key Learnings
See [SETUP.md](SETUP.md) for the full step-by-step installation guide with all pitfalls documented.
SETUP.md
+337
View File
@@ -0,0 +1,337 @@
# Full Setup Guide: ComfyUI + Trellis2 on Rocky Linux 9.x / RTX 4090
## Environment
- **OS:** Rocky Linux 9.7
- **GPU:** NVIDIA GeForce RTX 4090 (24 GB VRAM)
- **CUDA:** 12.8 (system toolkit)
- **CPU:** 128 GB RAM
- **Python:** 3.12
## Step 1: System Dependencies
### CUDA 12.8 Toolkit
Required for building sm_89 CUDA kernels from source.
```bash
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
sudo dnf install -y cuda-toolkit-12-8
```
### Eigen3 Headers (for CuMesh build)
```bash
sudo dnf install -y eigen3-devel
```
### Verify CUDA
```bash
/usr/local/cuda/bin/nvcc --version
# Should show: Cuda compilation tools, release 12.8
```
---
## Step 2: Virtual Environment
```bash
mkdir -p ~/Documents/ComfyUI_trellis2_pixal
cd ~/Documents/ComfyUI_trellis2_pixal
python3.12 -m venv venv
source venv/bin/activate
pip install --upgrade pip
```
---
## Step 3: PyTorch
```bash
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \
--index-url https://download.pytorch.org/whl/cu128
```
**⚡ Pitfall:** Do NOT install xformers yet. xformers ≥0.0.34 requires torch ≥2.10, but torchvision
is pinned to torch 2.9.1. Install xformers last with the correct version constraint.
---
## Step 4: ComfyUI
```bash
cd ~/Documents/ComfyUI_trellis2_pixal
git clone https://github.com/comfyanonymous/ComfyUI.git temp_comfy
mv temp_comfy/* temp_comfy/.[!.]* . 2>/dev/null
rmdir temp_comfy
```
Install ComfyUI dependencies:
```bash
pip install -r requirements.txt
```
---
## Step 5: Trellis2 Custom Nodes
```bash
cd custom_nodes
git clone https://github.com/visualbruno/ComfyUI-Trellis2.git
cd ..
```
Install Trellis2 Python dependencies:
```bash
pip install meshlib requests pymeshlab opencv-python scipy open3d plotly rembg
```
---
## Step 6: The CUDA Architecture Problem
**⚠️ CRITICAL PITFALL:** Almost every prebuilt CUDA wheel in the Trellis2 ecosystem
(flex_gemm, o_voxel, cumesh, nvdiffrast) is compiled **only for sm_120 (Blackwell)**.
The RTX 4090 is **sm_89 (Ada Lovelace)**. Using prebuilt wheels causes:
```
CUDA error: no kernel image is available for execution on the device
```
**Solution:** Rebuild all CUDA extension packages from source with `TORCH_CUDA_ARCH_LIST="8.9"`.
Before each source build, ensure CUDA is in PATH:
```bash
export PATH=/usr/local/cuda/bin:$PATH
export TORCH_CUDA_ARCH_LIST="8.9"
```
---
## Step 7: Source Builds (in order)
### 7a. utils3d — The Right GitHub Fork
**⚡ Pitfall:** `pip install utils3d` installs an empty placeholder package (v0.1.3) without
the `utils3d.pt` submodule that MoGe needs.
```bash
pip install "git+https://github.com/EasternJournalist/utils3d.git@3fab839f0be9931dac7c8488eb0e1600c236e183"
# Results in: utils3d-1.3
```
### 7b. flex_gemm — JeffreyXiang/FlexGEMM
```bash
pip uninstall flex_gemm -y 2>/dev/null
pip install "git+https://github.com/JeffreyXiang/FlexGEMM.git" --no-build-isolation
# Results in: flex_gemm-1.0.0
```
**Verification:** `.so` file should be ~3.4 MB (the prebuilt sm_120-only wheel is smaller).
### 7c. o_voxel — visualbruno Fork (NOT upstream main!)
**⚡ Pitfall:** The upstream `microsoft/TRELLIS.2` main branch is missing
`tiled_flexible_dual_grid_to_mesh`. The ComfyUI-Trellis2 author maintains a fork with
the required functions.
```bash
pip uninstall o_voxel -y 2>/dev/null
pip install "git+https://github.com/visualbruno/TRELLIS.2.git#subdirectory=o-voxel" --no-build-isolation
# Results in: o_voxel-0.0.1
```
**Verification:**
```python
import o_voxel.convert
assert hasattr(o_voxel.convert, 'tiled_flexible_dual_grid_to_mesh')
```
### 7d. cumesh — visualbruno/CuMesh (needs Eigen)
**⚡ Pitfall:** The `remesh_narrow_band_dc_quad` function only exists in visualbruno's fork.
The upstream `JeffreyXiang/CuMesh` only has `remesh_narrow_band_dc` (no `_quad` variant).
Also needs Eigen3 headers. Set `CPLUS_INCLUDE_PATH`:
```bash
pip uninstall cumesh -y 2>/dev/null
CPLUS_INCLUDE_PATH=/usr/include/eigen3 pip install "git+https://github.com/visualbruno/CuMesh.git" --no-build-isolation
# Results in: cumesh-1.0
```
**Verification:**
```python
import cumesh.remeshing
assert hasattr(cumesh.remeshing, 'remesh_narrow_band_dc_quad')
```
### 7e. nvdiffrast — NVlabs/nvdiffrast
**⚡ Pitfall:** The pip prebuilt wheel lacks sm_89 kernels. Build from source:
```bash
pip uninstall nvdiffrast -y 2>/dev/null
pip install "git+https://github.com/NVlabs/nvdiffrast.git" --no-build-isolation
# Results in: nvdiffrast-0.4.0
```
---
## Step 8: Attention Backend (xformers)
**⚡ Pitfall:** xformers version hell! Here's what works:
```bash
pip install xformers --index-url https://download.pytorch.org/whl/cu128
# This resolves to xformers-0.0.33.post2 with torch 2.9.1
```
**Why not newer?**
- xformers 0.0.34 → requires torch ≥2.10, pulls torch 2.10 which breaks torchvision
- xformers 0.0.35 → requires torch ≥2.10, same conflict
If pip pulls the wrong torch version, force-install all together:
```bash
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 xformers \
--index-url https://download.pytorch.org/whl/cu128
```
---
## Step 9: nATTen — Prebuilt Wheel
**⚡ Pitfall:** Building natten from source requires CMake + CUDA compiler + ~2 GB of `/tmp` space
for build dependencies. Use the prebuilt wheel instead:
```bash
pip install "natten==0.21.5+torch290cu128" -f https://whl.natten.org
# Results in: natten-0.21.5+torch290cu128
```
---
## Step 10: transformers v5 API Fix
**⚡ Pitfall:** `transformers ≥5` changed the DINOv3 ViT model structure. The internal
`self.model.layer` attribute moved to `self.model.model.layer`. This causes:
```
AttributeError: 'DINOv3ViTModel' object has no attribute 'layer'
```
Three files need patching in `custom_nodes/ComfyUI-Trellis2/trellis2/`:
```bash
cd ~/Documents/ComfyUI_trellis2_pixal/custom_nodes/ComfyUI-Trellis2/trellis2
# Fix all occurrences at once
sed -i 's/self\.model\.layer/self.model.model.layer/g' \
modules/image_feature_extractor.py \
trainers/flow_matching/mixins/image_conditioned.py \
trainers/flow_matching/mixins/image_conditioned_proj.py
```
Reference: [ComfyUI-Trellis2 Issue #144](https://github.com/visualbruno/ComfyUI-Trellis2/issues/144)
---
## Step 11: Disk Space
**⚡ Pitfall:** Building packages from source fills `/tmp` with pip build artifacts
(especially torch copies, 2+ GB each). Rocky Linux 9 default root partition is 70 GB.
```bash
# Clean pip build artifacts periodically
rm -rf /tmp/pip-* /tmp/pip-build-env-*
```
---
## Step 12: Models
Download manually (too large for automation):
| Model | Destination | Size |
|-------|-------------|------|
| `dinov3-vitl16-pretrain-lvd1689m` | `models/facebook/dinov3-vitl16-pretrain-lvd1689m/` | ~1.2 GB |
| `TRELLIS.2-4B` | `models/trellis2/TRELLIS.2-4B/` | ~8 GB |
---
## Verification
Start ComfyUI and run the "Trellis2ShapeCascadeGenerator" workflow:
```bash
cd ~/Documents/ComfyUI_trellis2_pixal
source venv/bin/activate
python main.py --listen 127.0.0.1
```
Expected output:
```
[INFO] ComfyUI version: 0.24.0
[INFO] pytorch version: 2.9.1+cu128
[INFO] xformers version: 0.0.33.post2
[INFO] Device: cuda:0 NVIDIA GeForce RTX 4090
[INFO] Import times for custom nodes:
[INFO] 0.8 seconds: .../ComfyUI-Trellis2
```
A full generation pipeline should complete in ~130 seconds.
---
## Package Version Summary
```
torch==2.9.1+cu128
torchvision==0.24.1+cu128
torchaudio==2.9.1+cu128
xformers==0.0.33.post2
natten==0.21.5+torch290cu128
utils3d==1.3
flex_gemm==1.0.0
o_voxel==0.0.1
cumesh==1.0
nvdiffrast==0.4.0
transformers>=5.0.0
```
---
## Common Errors & Fixes
| Error | Cause | Fix |
|-------|-------|-----|
| `CUDA error: no kernel image is available` | Package compiled for sm_120 only | Rebuild from source with `TORCH_CUDA_ARCH_LIST="8.9"` |
| `No module named 'utils3d'` | Empty pip package | Install GitHub fork (Step 7a) |
| `utils3d has no attribute 'pt'` | Wrong utils3d package | Install GitHub fork (Step 7a) |
| `DINOv3ViTModel has no attribute 'layer'` | transformers v5 API change | Apply sed fix (Step 10) |
| `No module named 'flash_attn'` / `'xformers'` | Missing attention backend | Install xformers (Step 8) |
| `No module named 'natten'` | Missing neighborhood attention | Install prebuilt wheel (Step 9) |
| `No module named 'tiled_flexible_dual_grid_to_mesh'` | Wrong o_voxel source | Use visualbruno fork (Step 7c) |
| `No module named 'remesh_narrow_band_dc_quad'` | Wrong cumesh source | Use visualbruno fork (Step 7d) |
| `Cuda error: 209` in nvdiffrast | Wrong nvdiffrast arch | Build from source (Step 7e) |
| `No space left on device` | /tmp filled by pip builds | Clean /tmp (Step 11) |
| `No CMAKE_CUDA_COMPILER could be found` | nvcc not in PATH | `export PATH=/usr/local/cuda/bin:$PATH` |
| `fatal error: Eigen/Dense` | Missing Eigen headers | `dnf install eigen3-devel` + set CPLUS_INCLUDE_PATH |
| xformers/torch version conflict | xformers≥0.0.34 needs torch≥2.10 | Pin to xformers 0.0.33.post2 (Step 8) |
---
## Credits
- [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
- [ComfyUI-Trellis2](https://github.com/visualbruno/ComfyUI-Trellis2) by visualbruno
- [TRELLIS.2](https://github.com/microsoft/TRELLIS.2) by Microsoft
- [FlexGEMM](https://github.com/JeffreyXiang/FlexGEMM) by JeffreyXiang
- [NATTEN](https://github.com/SHI-Labs/NATTEN) by SHI-Labs
- [nvdiffrast](https://github.com/NVlabs/nvdiffrast) by NVIDIA
- [MoGe](https://github.com/microsoft/MoGe) by Microsoft
build_cuda_extensions.sh
+54
View File
@@ -0,0 +1,54 @@
#!/bin/bash
# Build all CUDA extension packages from source for sm_89 (RTX 4090)
# Must run inside the activated venv with CUDA toolkit installed
#
# Usage: source venv/bin/activate && bash build_cuda_extensions.sh
set -e
export PATH="/usr/local/cuda/bin:$PATH"
export TORCH_CUDA_ARCH_LIST="8.9"
echo "=== Building CUDA extensions for sm_89 (RTX 4090) ==="
echo ""
# 1. utils3d (MoGe dependency — GitHub fork, NOT pip package)
echo "[1/5] utils3d (EasternJournalist fork)..."
pip install --quiet "git+https://github.com/EasternJournalist/utils3d.git@3fab839f0be9931dac7c8488eb0e1600c236e183"
echo " ✓ done"
# 2. flex_gemm
echo "[2/5] flex_gemm..."
pip uninstall -y flex_gemm 2>/dev/null || true
pip install --quiet "git+https://github.com/JeffreyXiang/FlexGEMM.git" --no-build-isolation
echo " ✓ done"
# 3. o_voxel (visualbruno fork — has tiled_flexible_dual_grid_to_mesh)
echo "[3/5] o_voxel..."
pip uninstall -y o_voxel 2>/dev/null || true
pip install --quiet "git+https://github.com/visualbruno/TRELLIS.2.git#subdirectory=o-voxel" --no-build-isolation
echo " ✓ done"
# 4. cumesh (visualbruno fork — has remesh_narrow_band_dc_quad, needs Eigen)
echo "[4/5] cumesh..."
pip uninstall -y cumesh 2>/dev/null || true
CPLUS_INCLUDE_PATH="/usr/include/eigen3:$CPLUS_INCLUDE_PATH" \
pip install --quiet "git+https://github.com/visualbruno/CuMesh.git" --no-build-isolation
echo " ✓ done"
# 5. nvdiffrast
echo "[5/5] nvdiffrast..."
pip uninstall -y nvdiffrast 2>/dev/null || true
pip install --quiet "git+https://github.com/NVlabs/nvdiffrast.git" --no-build-isolation
echo " ✓ done"
echo ""
echo "=== Verification ==="
python -c "
import utils3d; print('utils3d:', utils3d.__version__)
import flex_gemm; print('flex_gemm: OK')
import o_voxel.convert; assert hasattr(o_voxel.convert, 'tiled_flexible_dual_grid_to_mesh'); print('o_voxel: OK (tiled_flexible_dual_grid_to_mesh found)')
import cumesh.remeshing; assert hasattr(cumesh.remeshing, 'remesh_narrow_band_dc_quad'); print('cumesh: OK (remesh_narrow_band_dc_quad found)')
import nvdiffrast.torch; print('nvdiffrast: OK (RasterizeCudaContext available)')
"
echo "=== All CUDA extensions built successfully ==="
fix_transformers_v5.sh
+33
View File
@@ -0,0 +1,33 @@
#!/bin/bash
# Apply transformers v5 API fix to ComfyUI-Trellis2
# Fixes: AttributeError: 'DINOv3ViTModel' object has no attribute 'layer'
# Reference: https://github.com/visualbruno/ComfyUI-Trellis2/issues/144
TRELLIS2_DIR="${1:-custom_nodes/ComfyUI-Trellis2/trellis2}"
if [ ! -d "$TRELLIS2_DIR" ]; then
echo "ERROR: Trellis2 directory not found: $TRELLIS2_DIR"
echo "Usage: $0 [path/to/ComfyUI-Trellis2/trellis2]"
exit 1
fi
echo "Patching DINOv3 model.layer → model.model.layer in:"
for f in \
modules/image_feature_extractor.py \
trainers/flow_matching/mixins/image_conditioned.py \
trainers/flow_matching/mixins/image_conditioned_proj.py
do
file="$TRELLIS2_DIR/$f"
if [ -f "$file" ]; then
if grep -q 'self\.model\.layer' "$file"; then
sed -i 's/self\.model\.layer/self.model.model.layer/g' "$file"
echo " ✓ $f"
else
echo " - $f (already patched)"
fi
else
echo " ✗ $f (not found)"
fi
done
echo "Done."
requirements.txt
+29
View File
@@ -0,0 +1,29 @@
# ComfyUI base (from ComfyUI/requirements.txt)
# Install with: pip install -r requirements_comfyui.txt
-r https://raw.githubusercontent.com/comfyanonymous/ComfyUI/master/requirements.txt
# Trellis2 Python dependencies
meshlib
requests
pymeshlab
opencv-python
scipy
open3d
plotly
rembg
# Core
torch==2.9.1
torchvision==0.24.1
torchaudio==2.9.1
# Attention backends
xformers==0.0.33.post2
natten==0.21.5+torch290cu128
# Source-built CUDA extensions (use pip install from GitHub, see SETUP.md)
# flex_gemm @ git+https://github.com/JeffreyXiang/FlexGEMM.git
# o_voxel @ git+https://github.com/visualbruno/TRELLIS.2.git#subdirectory=o-voxel
# cumesh @ git+https://github.com/visualbruno/CuMesh.git
# nvdiffrast @ git+https://github.com/NVlabs/nvdiffrast.git
# utils3d @ git+https://github.com/EasternJournalist/utils3d.git@3fab839