comfyui-trellis2-rocky-setup/SETUP.md

# Full Setup Guide: ComfyUI + Trellis2 on Rocky Linux 9.x / RTX 4090

## Environment

- **OS:** Rocky Linux 9.7
- **GPU:** NVIDIA GeForce RTX 4090 (24 GB VRAM)
- **CUDA:** 12.8 (system toolkit)
- **CPU:** 128 GB RAM
- **Python:** 3.12

## Step 1: System Dependencies

### CUDA 12.8 Toolkit

Required for building sm_89 CUDA kernels from source.

```bash
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
sudo dnf install -y cuda-toolkit-12-8
```

### Eigen3 Headers (for CuMesh build)

```bash
sudo dnf install -y eigen3-devel
```

### Verify CUDA

```bash
/usr/local/cuda/bin/nvcc --version
# Should show: Cuda compilation tools, release 12.8
```

---

## Step 2: Virtual Environment

```bash
mkdir -p ~/Documents/ComfyUI_trellis2_pixal
cd ~/Documents/ComfyUI_trellis2_pixal
python3.12 -m venv venv
source venv/bin/activate
pip install --upgrade pip
```

---

## Step 3: PyTorch

```bash
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \
  --index-url https://download.pytorch.org/whl/cu128
```

**⚡ Pitfall:** Do NOT install xformers yet. xformers ≥0.0.34 requires torch ≥2.10, but torchvision
is pinned to torch 2.9.1. Install xformers last with the correct version constraint.

---

## Step 4: ComfyUI

```bash
cd ~/Documents/ComfyUI_trellis2_pixal
git clone https://github.com/comfyanonymous/ComfyUI.git temp_comfy
mv temp_comfy/* temp_comfy/.[!.]* . 2>/dev/null
rmdir temp_comfy
```

Install ComfyUI dependencies:

```bash
pip install -r requirements.txt
```

---

## Step 5: Trellis2 Custom Nodes

```bash
cd custom_nodes
git clone https://github.com/visualbruno/ComfyUI-Trellis2.git
cd ..
```

Install Trellis2 Python dependencies:

```bash
pip install meshlib requests pymeshlab opencv-python scipy open3d plotly rembg
```

---

## Step 6: The CUDA Architecture Problem

**⚠️ CRITICAL PITFALL:** Almost every prebuilt CUDA wheel in the Trellis2 ecosystem
(flex_gemm, o_voxel, cumesh, nvdiffrast) is compiled **only for sm_120 (Blackwell)**.
The RTX 4090 is **sm_89 (Ada Lovelace)**. Using prebuilt wheels causes:

```
CUDA error: no kernel image is available for execution on the device
```

**Solution:** Rebuild all CUDA extension packages from source with `TORCH_CUDA_ARCH_LIST="8.9"`.

Before each source build, ensure CUDA is in PATH:

```bash
export PATH=/usr/local/cuda/bin:$PATH
export TORCH_CUDA_ARCH_LIST="8.9"
```

---

## Step 7: Source Builds (in order)

### 7a. utils3d — The Right GitHub Fork

**⚡ Pitfall:** `pip install utils3d` installs an empty placeholder package (v0.1.3) without
the `utils3d.pt` submodule that MoGe needs.

```bash
pip install "git+https://github.com/EasternJournalist/utils3d.git@3fab839f0be9931dac7c8488eb0e1600c236e183"
# Results in: utils3d-1.3
```

### 7b. flex_gemm — JeffreyXiang/FlexGEMM

```bash
pip uninstall flex_gemm -y 2>/dev/null
pip install "git+https://github.com/JeffreyXiang/FlexGEMM.git" --no-build-isolation
# Results in: flex_gemm-1.0.0
```

**Verification:** `.so` file should be ~3.4 MB (the prebuilt sm_120-only wheel is smaller).

### 7c. o_voxel — visualbruno Fork (NOT upstream main!)

**⚡ Pitfall:** The upstream `microsoft/TRELLIS.2` main branch is missing
`tiled_flexible_dual_grid_to_mesh`. The ComfyUI-Trellis2 author maintains a fork with
the required functions.

```bash
pip uninstall o_voxel -y 2>/dev/null
pip install "git+https://github.com/visualbruno/TRELLIS.2.git#subdirectory=o-voxel" --no-build-isolation
# Results in: o_voxel-0.0.1
```

**Verification:**
```python
import o_voxel.convert
assert hasattr(o_voxel.convert, 'tiled_flexible_dual_grid_to_mesh')
```

### 7d. cumesh — visualbruno/CuMesh (needs Eigen)

**⚡ Pitfall:** The `remesh_narrow_band_dc_quad` function only exists in visualbruno's fork.
The upstream `JeffreyXiang/CuMesh` only has `remesh_narrow_band_dc` (no `_quad` variant).

Also needs Eigen3 headers. Set `CPLUS_INCLUDE_PATH`:

```bash
pip uninstall cumesh -y 2>/dev/null
CPLUS_INCLUDE_PATH=/usr/include/eigen3 pip install "git+https://github.com/visualbruno/CuMesh.git" --no-build-isolation
# Results in: cumesh-1.0
```

**Verification:**
```python
import cumesh.remeshing
assert hasattr(cumesh.remeshing, 'remesh_narrow_band_dc_quad')
```

### 7e. nvdiffrast — NVlabs/nvdiffrast

**⚡ Pitfall:** The pip prebuilt wheel lacks sm_89 kernels. Build from source:

```bash
pip uninstall nvdiffrast -y 2>/dev/null
pip install "git+https://github.com/NVlabs/nvdiffrast.git" --no-build-isolation
# Results in: nvdiffrast-0.4.0
```

---

## Step 8: Attention Backend (xformers)

**⚡ Pitfall:** xformers version hell! Here's what works:

```bash
pip install xformers --index-url https://download.pytorch.org/whl/cu128
# This resolves to xformers-0.0.33.post2 with torch 2.9.1
```

**Why not newer?**
- xformers 0.0.34 → requires torch ≥2.10, pulls torch 2.10 which breaks torchvision
- xformers 0.0.35 → requires torch ≥2.10, same conflict

If pip pulls the wrong torch version, force-install all together:
```bash
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 xformers \
  --index-url https://download.pytorch.org/whl/cu128
```

---

## Step 9: nATTen — Prebuilt Wheel

**⚡ Pitfall:** Building natten from source requires CMake + CUDA compiler + ~2 GB of `/tmp` space
for build dependencies. Use the prebuilt wheel instead:

```bash
pip install "natten==0.21.5+torch290cu128" -f https://whl.natten.org
# Results in: natten-0.21.5+torch290cu128
```

---

## Step 10: transformers v5 API Fix

**⚡ Pitfall:** `transformers ≥5` changed the DINOv3 ViT model structure. The internal
`self.model.layer` attribute moved to `self.model.model.layer`. This causes:

```
AttributeError: 'DINOv3ViTModel' object has no attribute 'layer'
```

Three files need patching in `custom_nodes/ComfyUI-Trellis2/trellis2/`:

```bash
cd ~/Documents/ComfyUI_trellis2_pixal/custom_nodes/ComfyUI-Trellis2/trellis2

# Fix all occurrences at once
sed -i 's/self\.model\.layer/self.model.model.layer/g' \
  modules/image_feature_extractor.py \
  trainers/flow_matching/mixins/image_conditioned.py \
  trainers/flow_matching/mixins/image_conditioned_proj.py
```

Reference: [ComfyUI-Trellis2 Issue #144](https://github.com/visualbruno/ComfyUI-Trellis2/issues/144)

---

## Step 11: Disk Space

**⚡ Pitfall:** Building packages from source fills `/tmp` with pip build artifacts
(especially torch copies, 2+ GB each). Rocky Linux 9 default root partition is 70 GB.

```bash
# Clean pip build artifacts periodically
rm -rf /tmp/pip-* /tmp/pip-build-env-*
```

---

## Step 12: Models

Download manually (too large for automation):

| Model | Destination | Size |
|-------|-------------|------|
| `dinov3-vitl16-pretrain-lvd1689m` | `models/facebook/dinov3-vitl16-pretrain-lvd1689m/` | ~1.2 GB |
| `TRELLIS.2-4B` | `models/trellis2/TRELLIS.2-4B/` | ~8 GB |

---

## Verification

Start ComfyUI and run the "Trellis2ShapeCascadeGenerator" workflow:

```bash
cd ~/Documents/ComfyUI_trellis2_pixal
source venv/bin/activate
python main.py --listen 127.0.0.1
```

Expected output:
```
[INFO] ComfyUI version: 0.24.0
[INFO] pytorch version: 2.9.1+cu128
[INFO] xformers version: 0.0.33.post2
[INFO] Device: cuda:0 NVIDIA GeForce RTX 4090
[INFO] Import times for custom nodes:
[INFO]    0.8 seconds: .../ComfyUI-Trellis2
```

A full generation pipeline should complete in ~130 seconds.

---

## Package Version Summary

```
torch==2.9.1+cu128
torchvision==0.24.1+cu128
torchaudio==2.9.1+cu128
xformers==0.0.33.post2
natten==0.21.5+torch290cu128
utils3d==1.3
flex_gemm==1.0.0
o_voxel==0.0.1
cumesh==1.0
nvdiffrast==0.4.0
transformers>=5.0.0
```

---

## Common Errors & Fixes

| Error | Cause | Fix |
|-------|-------|-----|
| `CUDA error: no kernel image is available` | Package compiled for sm_120 only | Rebuild from source with `TORCH_CUDA_ARCH_LIST="8.9"` |
| `No module named 'utils3d'` | Empty pip package | Install GitHub fork (Step 7a) |
| `utils3d has no attribute 'pt'` | Wrong utils3d package | Install GitHub fork (Step 7a) |
| `DINOv3ViTModel has no attribute 'layer'` | transformers v5 API change | Apply sed fix (Step 10) |
| `No module named 'flash_attn'` / `'xformers'` | Missing attention backend | Install xformers (Step 8) |
| `No module named 'natten'` | Missing neighborhood attention | Install prebuilt wheel (Step 9) |
| `No module named 'tiled_flexible_dual_grid_to_mesh'` | Wrong o_voxel source | Use visualbruno fork (Step 7c) |
| `No module named 'remesh_narrow_band_dc_quad'` | Wrong cumesh source | Use visualbruno fork (Step 7d) |
| `Cuda error: 209` in nvdiffrast | Wrong nvdiffrast arch | Build from source (Step 7e) |
| `No space left on device` | /tmp filled by pip builds | Clean /tmp (Step 11) |
| `No CMAKE_CUDA_COMPILER could be found` | nvcc not in PATH | `export PATH=/usr/local/cuda/bin:$PATH` |
| `fatal error: Eigen/Dense` | Missing Eigen headers | `dnf install eigen3-devel` + set CPLUS_INCLUDE_PATH |
| xformers/torch version conflict | xformers≥0.0.34 needs torch≥2.10 | Pin to xformers 0.0.33.post2 (Step 8) |

---

## Credits

- [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
- [ComfyUI-Trellis2](https://github.com/visualbruno/ComfyUI-Trellis2) by visualbruno
- [TRELLIS.2](https://github.com/microsoft/TRELLIS.2) by Microsoft
- [FlexGEMM](https://github.com/JeffreyXiang/FlexGEMM) by JeffreyXiang
- [NATTEN](https://github.com/SHI-Labs/NATTEN) by SHI-Labs
- [nvdiffrast](https://github.com/NVlabs/nvdiffrast) by NVIDIA
- [MoGe](https://github.com/microsoft/MoGe) by Microsoft