Skip to content

Port Ideogram 4.0 to JAX / MaxDiffusion#442

Open
mbohlool wants to merge 1 commit into
mainfrom
ideogram_jax
Open

Port Ideogram 4.0 to JAX / MaxDiffusion#442
mbohlool wants to merge 1 commit into
mainfrom
ideogram_jax

Conversation

@mbohlool

@mbohlool mbohlool commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Title: Port Ideogram 4.0 to JAX / MaxDiffusion

Summary:
This PR ports the Ideogram 4.0 pipeline to JAX. It introduces the native Flax NNX Ideogram4Transformer, integrates the TorchaxQwen3VLTextEncoder for complex prompt processing, and implements the required asymmetric CFG denoising loop. It also includes caption key re-ordering in the pipeline to enforce Ideogram's strict JSON-based prompt schema (high_level_description, style_description, compositional_deconstruction).

Generated Examples:

Prompt:

{
  "high_level_description": "A sleek red sports car driving through a futuristic cyberpunk city in the rain, with neon reflections on the wet asphalt.",
  "style_description": {
    "aesthetics": "cyberpunk, futuristic, moody, dynamic",
    "lighting": "neon lights, cinematic lighting, reflections",
    "photo": "photorealistic, motion blur on background",
    "medium": "digital photography",
    "color_palette": ["#E74C3C", "#3498DB", "#9B59B6", "#1ABC9C", "#2C3E50"]
  },
  "compositional_deconstruction": {
    "background": "A bustling cyberpunk cityscape at night, towering skyscrapers, heavy rain.",
    "elements": [
      {
        "type": "obj",
        "bbox": [400, 100, 900, 900],
        "desc": "A highly detailed, sleek red futuristic sports car speeding forward."
      },
      {
        "type": "obj",
        "bbox": [800, 0, 1000, 1000],
        "desc": "Wet asphalt road reflecting vibrant pink and blue neon lights."
      }
    ]
  }
}
pr_magic_car_ideogram_output_42_0

Prompt:

{
  "high_level_description": "A ginger cat wearing a tiny wizard hat reading a glowing spellbook.",
  "style_description": {
    "aesthetics": "magical, fantasy, whimsical",
    "lighting": "warm glowing light, magical illumination from the book",
    "photo": "highly detailed, macro photography, shallow depth of field",
    "medium": "digital photography, fantasy illustration",
    "color_palette": ["#E67E22", "#8E44AD", "#F1C40F", "#2C3E50", "#D35400"]
  },
  "compositional_deconstruction": {
    "background": "A dimly lit, dusty wizard's study filled with old scrolls and potions.",
    "elements": [
      {
        "type": "obj",
        "bbox": [100, 200, 800, 800],
        "desc": "A fluffy ginger cat wearing a detailed purple wizard hat with gold stars."
      },
      {
        "type": "obj",
        "bbox": [700, 200, 1000, 800],
        "desc": "An ancient, open spellbook glowing with magical yellow light."
      }
    ]
  }
}
pr_magic_cat_ideogram_output_42_0

Prompt:

{
  "high_level_description": "A stunning cinematic shot of a futuristic city at night with flying cars and bright neon lights.",
  "style_description": {
    "aesthetics": "cinematic, futuristic, sci-fi",
    "lighting": "neon lighting, night time, glowing cityscapes",
    "photo": "photorealistic, hyper-detailed, wide angle",
    "medium": "digital photography",
    "color_palette": ["#00FFFF", "#FF00FF", "#000080", "#FFD700", "#191970"]
  },
  "compositional_deconstruction": {
    "background": "A dark night sky overlooking a sprawling futuristic megalopolis.",
    "elements": [
      {
        "type": "obj",
        "bbox": [200, 0, 1000, 1000],
        "desc": "Dense cityscape with towering skyscrapers illuminated by intricate neon signs."
      },
      {
        "type": "obj",
        "bbox": [100, 200, 400, 800],
        "desc": "Several futuristic flying cars with glowing thrusters streaking across the sky."
      }
    ]
  }
}
pr_magic_city_ideogram_output_42_0

@mbohlool mbohlool requested a review from entrpn as a code owner July 8, 2026 21:37
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

@mbohlool mbohlool force-pushed the ideogram_jax branch 8 times, most recently from 83fcf30 to 23fa1c2 Compare July 8, 2026 23:27
…r and Qwen text encoder

- Implements `Ideogram4Transformer` using `flax.nnx`
- Ports `TorchaxQwen3VLTextEncoder` to handle Ideogram's LLM prompt conditions
- Adds strict JSON schema caption key re-ordering in `IdeogramPipeline`
- Adds full asymmetric CFG denoising loop
@mbohlool

mbohlool commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

/gemini review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant