Fix incorrect sharding for TSP, DP, fused_mlp workloads

[dinodeep]

Description

When performing tensor sequence parallelism and data parallelism with fused_mlp on Llama models, the up projection activations end up being sharded on the sequence dimension when they are supposed to be sharded on the hidden dimension. This is because fused_mlp adds an additional dimension ("num_activations") that goes unaccounted for, offsetting the sharding.

%dot_general.554 = bf16[8,8192,2,28672]{3,2,1,0} dot(...) ... sharding={devices=[2,4,1,1]<=[8]},

This leads to poorer E2E step time as it introduces exposed all-to-all communication operations which could have been avoided with more constrained sharding annotations. This PR resolves this by adding in a constraint for the fused_mlp case to ensure that the intermediate activations are sharded appropriately as shown below.

%dot_general.66 = bf16[8,8192,2,28672]{3,2,1,0} dot(...), ..., sharding={devices=[2,1,1,4]<=[8]}

Tests

Tested this change by running a small E2E training run that hit this error and observed performance improvement. (Same memory consumption and compile times)

This is performance before the fix:

completed step: 15, seconds: 0.289, TFLOP/s/device: 962.202, Tokens/s/device: 28311.342, total_weights: 65536, loss: 0.187, lm_loss: 0.187, perplexity: 1.205
completed step: 16, seconds: 0.286, TFLOP/s/device: 972.357, Tokens/s/device: 28610.145, total_weights: 65536, loss: 0.164, lm_loss: 0.164, perplexity: 1.178
completed step: 17, seconds: 0.298, TFLOP/s/device: 934.486, Tokens/s/device: 27495.838, total_weights: 65536, loss: 0.151, lm_loss: 0.151, perplexity: 1.163
completed step: 18, seconds: 0.294, TFLOP/s/device: 946.926, Tokens/s/device: 27861.861, total_weights: 65536, loss: 0.142, lm_loss: 0.142, perplexity: 1.153
completed step: 19, seconds: 0.288, TFLOP/s/device: 968.047, Tokens/s/device: 28483.312, total_weights: 65536, loss: 0.136, lm_loss: 0.136, perplexity: 1.146
completed step: 20, seconds: 0.295, TFLOP/s/device: 942.974, Tokens/s/device: 27745.602, total_weights: 65536, loss: 0.131, lm_loss: 0.131, perplexity: 1.140

This is the performance after the fix:

completed step: 15, seconds: 0.272, TFLOP/s/device: 1025.273, Tokens/s/device: 30167.112, total_weights: 65536, loss: 0.187, lm_loss: 0.187, perplexity: 1.206
completed step: 16, seconds: 0.268, TFLOP/s/device: 1038.385, Tokens/s/device: 30552.914, total_weights: 65536, loss: 0.165, lm_loss: 0.165, perplexity: 1.179
completed step: 17, seconds: 0.277, TFLOP/s/device: 1005.922, Tokens/s/device: 29597.728, total_weights: 65536, loss: 0.151, lm_loss: 0.151, perplexity: 1.163
completed step: 18, seconds: 0.278, TFLOP/s/device: 1000.593, Tokens/s/device: 29440.938, total_weights: 65536, loss: 0.143, lm_loss: 0.143, perplexity: 1.153
completed step: 19, seconds: 0.274, TFLOP/s/device: 1014.602, Tokens/s/device: 29853.139, total_weights: 65536, loss: 0.137, lm_loss: 0.137, perplexity: 1.146
completed step: 20, seconds: 0.274, TFLOP/s/device: 1015.487, Tokens/s/device: 29879.163, total_weights: 65536, loss: 0.132, lm_loss: 0.132, perplexity: 1.141

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!