# Predict function ignore parameters

**URL:** <https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192>\
**Category:** Amazon SageMaker\
**Created:** [January 28, 2022, 10:15am UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192 "2022-01-28T10:15:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![AliYZ](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@AliYZ](https://discuss.huggingface.co/u/AliYZ)\
**Post date:** [January 28, 2022, 10:15am UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/1 "2022-01-28T10:15:23Z")

</div>

Hey,  
I’m trying to deploy a Hugging Face Model (GPT-Neo) on the SageMaker endpoint. I followed the official [example](https://github.com/huggingface/notebooks/blob/master/sagemaker/11_deploy_model_from_hf_hub/deploy_transformer_model_from_hf_hub.ipynb) and this [forum](https://discuss.huggingface.co/t/inference-hyperparameters/9539). but it seems that the generate function is totally ignoring my parameters (it generates just one word despite setting the min length to 10000!) Any idea what is wrong?

My code:

```python
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'EleutherAI/gpt-neo-1.3B',
	'HF_TASK':'text-generation'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.g4dn.xlarge' # ec2 instance type
)

prompt = "Some prompt"

gen_tex = predictor.predict({
	"inputs": prompt,
    "parameters" : {"min_length":10,}
})

print(gen_tex[0]['generated_text'])

```

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [January 28, 2022, 10:45am UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/2 "2022-01-28T10:45:26Z")

</div>

Could you please update to the latest version? which would be `transformers_version="4.12"` and `pytorch_version="1.9"` and test it again?

What is the output of `gen_tex[0]['generated_text']`? Is the `,` in the “parameters” on purpose?

---

<div class="post-metadata">

**Author:** ![AliYZ](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@AliYZ](https://discuss.huggingface.co/u/AliYZ)\
**Post date:** [January 28, 2022, 11:20am UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/3 "2022-01-28T11:20:27Z")

</div>

Updated to the latest version but it still ignore the parameter:

```python
predictor.predict({
    'inputs': "Can you please let us know more details about your",
    'parameters': {"min_length":1000}
})

```

output:

````python
[{'generated_text': 'Can you please let us know more details about your account?\n\nHello,\nI am interested in the above-mentioned company and I have read some very interesting articles about it. I am interested in starting the work. Please let me know if'}]```
````

---

<div class="post-metadata">

**Author:** ![AliYZ](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@AliYZ](https://discuss.huggingface.co/u/AliYZ)\
**Post date:** [January 28, 2022, 11:24am UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/4 "2022-01-28T11:24:13Z")

</div>

I guess it’s only for min\_length since it works fine for max\_length=3  
output:

```python
[{'generated_text': 'Can you please let us know more details about your research'}]

```

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [January 28, 2022, 12:24pm UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/5 "2022-01-28T12:24:22Z")

</div>

And when you run `model.generate` with your parameter in a colab or other environment with the same `Transformers` version it works?

---

<div class="post-metadata">

**Author:** ![AliYZ](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@AliYZ](https://discuss.huggingface.co/u/AliYZ)\
**Post date:** [January 28, 2022, 12:37pm UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/6 "2022-01-28T12:37:29Z")

</div>

Yes, it works fine when I use it locally with `from_pretrained` model.

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [January 28, 2022, 12:57pm UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/7 "2022-01-28T12:57:48Z")

</div>

Hi Ali, what happens if you set the `min_length` and the `max_length` parameters explicitly? I’m asking because I believe the text generator uses the `max_length` parameter from the model configuration if you don’t set it explicitly. And if the `max_length` parameter in the config file is smaller than you `min_length` the output would be truncated. So, just wondering if setting both (e.g. `min_length=1000` and `max_length=2000`) helps?

---

<div class="post-metadata">

**Author:** ![AliYZ](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@AliYZ](https://discuss.huggingface.co/u/AliYZ)\
**Post date:** [January 28, 2022, 1:29pm UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/8 "2022-01-28T13:29:51Z")

</div>

Thank you both for your help. That solved the problem @marshmellow77.

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [January 28, 2022, 1:46pm UTC](https://discuss.huggingface.co/t/predict-function-ignore-parameters/14192/9 "2022-01-28T13:46:42Z")

</div>

Awesome, glad it worked. It’d be great if you could mark this thread as Answered/Solved - it would make it easier and quicker in the future for other users with the same problem to find the solution 🙂

Cheers  
Heiko