# CLIPVisionModel Padding Problem

**URL:** <https://discuss.huggingface.co/t/clipvisionmodel-padding-problem/124187>\
**Category:** 🤗Transformers\
**Created:** [November 16, 2024, 6:33pm UTC](https://discuss.huggingface.co/t/clipvisionmodel-padding-problem/124187 "2024-11-16T18:33:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![yunust](https://avatars.discourse-cdn.com/v4/letter/y/a3d4f5/32.png) [@yunust](https://discuss.huggingface.co/u/yunust)\
**Post date:** [November 16, 2024, 6:33pm UTC](https://discuss.huggingface.co/t/clipvisionmodel-padding-problem/124187/1 "2024-11-16T18:33:25Z")

</div>

The documentation on CLIPVIsionModel says that :

> Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using[AutoImageProcessor]

However, when I pad my image with image\_transforms.pad and do a forward pass, results become much different than the normal image.

```auto
from transformers.image_transforms import pad
import numpy as np

# Example image as a NumPy array
image = np.random.rand(224, 224, 3) # Height x Width x Channels

# Define padding: ((before_height, after_height), (before_width, after_width))
padding = ((0, 0), (112, 112)) # Pads width to make it 448

# Apply padding
padded_image = pad(image, padding=padding)
print("Original Image Shape:", image.shape)
print("Padded Image Shape:", padded_image.shape)

image_torch = torch.tensor(image).permute(2, 0, 1).unsqueeze(0)
padded_image_torch = torch.tensor(padded_image).permute(2, 0, 1).unsqueeze(0)

print("Original Image Shape (Torch):", image_torch.shape)
print("Padded Image Shape (Torch):", padded_image_torch.shape)
# Step 5: Pass the padded image through the model
outputs_padded = model(pixel_values=padded_image_torch, interpolate_pos_encoding=True)
outputs_original = model(pixel_values=image_torch)
# Step 6: Extract the results for comparison
original = outputs_original.pooler_output
padded = outputs_padded.pooler_output

print(torch.mean(original - padded))

```

how to handle image padding properly.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 17, 2024, 2:43am UTC](https://discuss.huggingface.co/t/clipvisionmodel-padding-problem/124187/2 "2024-11-17T02:43:47Z")

</div>

It’s a bug, no matter how we look at it. Please see this behavior.😭  
It looks like we should submit an issue or PR to github…

```py
from transformers.image_transforms import pad
import numpy as np
import torch
from PIL import Image
from transformers import CLIPImageProcessor, CLIPVisionModel
model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Example image as a NumPy array
image = np.random.rand(224, 224, 3) # Height x Width x Channels
image_pil = np.array(Image.fromarray(image, 'RGB')) # Open with PIL and save

# Define padding: ((before_height, after_height), (before_width, after_width))
padding = ((0, 0), (112, 112)) # Pads width to make it 448

# Apply padding
padded_image = pad(image, padding=padding)
padded_image_pil = pad(image_pil, padding=padding)
print("Original Image Shape:", image.shape)
print("Padded Image Shape:", padded_image.shape)

image_torch = torch.tensor(image).permute(2, 0, 1).unsqueeze(0)
padded_image_torch = torch.tensor(padded_image).permute(2, 0, 1).unsqueeze(0)

print("Original Image Shape (Torch):", image_torch.shape)
print("Padded Image Shape (Torch):", padded_image_torch.shape)
# Step 5: Pass the padded image through the model
outputs_padded = model(pixel_values=padded_image_torch, interpolate_pos_encoding=True)
outputs_original = model(pixel_values=image_torch)
# Step 6: Extract the results for comparison
original = outputs_original.pooler_output
padded = outputs_padded.pooler_output

print(torch.mean(original - padded))

# Save images
original_im = Image.fromarray(image, 'RGB')
padded_im = Image.fromarray(padded_image, 'RGB')
padded_im_pil = Image.fromarray(padded_image_pil, 'RGB')
original_im.save("_pad_original.png") # normal
padded_im.save("_pad_padded.png") # strange
padded_im_pil.save("_pad_padded_pil.png") # normal

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 18, 2024, 10:58am UTC](https://discuss.huggingface.co/t/clipvisionmodel-padding-problem/124187/3 "2024-11-18T10:58:24Z")

</div>

I opened issue.

> <https://github.com/huggingface/transformers/issues/34775>
>
> \### System Info
> 
> \- \`transformers\` version: 4.46.1
> \- Platform: Windows-10-10.0.1…9045-SP0
> \- Python version: 3.9.13
> \- Huggingface\_hub version: 0.26.2
> \- Safetensors version: 0.4.3
> \- Accelerate version: 1.0.1
> \- Accelerate config: not found
> \- PyTorch version (GPU?): 2.4.0+cu124 (True)
> \- Tensorflow version (GPU?): not installed (NA)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> \- Using distributed or parallel set-up in script?: \<fill in\>
> \- Using GPU in script?: \<fill in\>
> \- GPU type: NVIDIA GeForce RTX 3060 Ti
> 
> \### Who can help?
> 
> \_No response\_
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[X\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[X\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> \`\`\`py
> from transformers.image\_transforms import pad
> import numpy as np
> import torch
> from PIL import Image
> 
> \# Example image as a NumPy array
> image = np.random.rand(224, 224, 3) # Height x Width x Channels
> image\_pil = np.array(Image.fromarray(image, 'RGB')) # Open with PIL and save
> image\_uint8 = (image \* 255.0).astype(np.uint8)
> 
> \# Define padding: ((before\_height, after\_height), (before\_width, after\_width))
> padding = ((0, 0), (112, 112)) # Pads width to make it 448
> 
> \# Apply padding
> padded\_image = pad(image, padding=padding)
> padded\_image\_pil = pad(image\_pil, padding=padding)
> padded\_image\_uint8 = pad(image\_uint8, padding=padding)
> print("Original Image Shape:", image.shape)
> print("Padded Image Shape:", padded\_image.shape)
> print("Padded Image Shape (PIL):", padded\_image\_pil.shape)
> print("Padded Image Shape (uint8):", padded\_image\_uint8.shape)
> 
> image\_torch = torch.tensor(image).permute(2, 0, 1).unsqueeze(0)
> padded\_image\_torch = torch.tensor(padded\_image).permute(2, 0, 1).unsqueeze(0)
> padded\_image\_pil\_torch = torch.tensor(padded\_image\_pil).permute(2, 0, 1).unsqueeze(0)
> padded\_image\_uint8\_torch = torch.tensor(padded\_image\_uint8).permute(2, 0, 1).unsqueeze(0)
> 
> print("Original Image Shape (Torch):", image\_torch.shape)
> print("Padded Image Shape (Torch):", padded\_image\_torch.shape)
> print("Padded Image Shape (PIL) (Torch):", padded\_image\_pil\_torch.shape)
> print("Padded Image Shape (uint8) (Torch):", padded\_image\_uint8\_torch.shape)
> 
> \# Save images
> original\_im = Image.fromarray(image, 'RGB')
> padded\_im = Image.fromarray(padded\_image, 'RGB')
> padded\_im\_pil = Image.fromarray(padded\_image\_pil, 'RGB')
> padded\_im\_uint8 = Image.fromarray(padded\_image\_uint8, 'RGB')
> original\_im.save("\_pad\_original.png") # normal
> padded\_im.save("\_pad\_padded.png") # strange
> padded\_im\_pil.save("\_pad\_padded\_pil.png") # normal
> padded\_im\_uint8.save("\_pad\_padded\_uint8.png") # relatively normal
> \`\`\`
> 
> \### Expected behavior
> 
> After receiving a report on the Hugging Face forum that the \`padding\` in the \`transformers\` library was behaving strangely, I investigated and found the approximate cause.
> It seems that the \`pad\` function in \`numpy\` returns strange results when it receives an ndarray that is not \`uint8\`.
> As a simple workaround, there is a method of converting it to \`Pillow Image\` once, but this method is dependent on \`Pillow\`.
> If the library converts it to \`uint8\` on its own, it may be a little troublesome to judge the numerical range of the image.
> I opened an issue instead of a PR because I couldn't think of a good implementation.
> 
> \### Reference
> https://discuss.huggingface.co/t/clipvisionmodel-padding-problem/124187 A post that helps me find a problem
> https://discord.com/channels/879548962464493619/1301002234963820604 Investigating preprocessor bugs
> \### Dependencies
> \`\`\`
> transformers==4.46.2
> torch==2.4.0
> numpy\<2
> \`\`\`
> \### Demo
> https://huggingface.co/spaces/John6666/transformers\_padding\_bug\_test