Moderation not working properly, leads to model distortion

[AnMaJ]

I was experimenting with your work: Moderator. I tried working with it by running the code provided for Moderator on your github: https://github.com/DataSmithLab/Moderator/tree/main. I tested with a very basic policy of removing Naked Women, or Drugs, or Replacing Donald Trump with a Duck, with finetuning the model with around 400 images(I have kept the scaling value as 88/100 and used descriptive prompt expansion).

Unfortunately, when I try to generate the images using the edited model, the generated images are very distorted and unstructured. Additionally, for the replacement policy, where I replace Donald Trump with Duck, there is no observable replacement, and the images turn out to be even more distorted. I am sharing a few of the images generated by the edited (stable diffusion xl) model here for your kind reference. I went through the entire codebase, and there seems to be no problem with the implementation, i.e. it completely aligns with the paper, and so does the flow of the policy enforcement which I try to run.

Therefore, I request you to kindly guide me with potential issues I might be running into while trying to run your pipeline.

Thank you!

[haojian]

This might be some basic LoRA fine-tuning problems. It seems the weight of the task vectors is too high, so it breaks the original model.

You may debug it by running some isolated fine-tuning ablation experiments. Get some images, fine-tune the models, and see if your fine-tuning works. Once the fine-tuning part is stable, you can integrate it into the Moderator pipeline.