I tried to run ’scripts/run_biorex_exp.sh‘ to reproduce your results, but ran into the following trouble:
When I test directly with microsoft pre-trained models, the warning prompts:

[INFO|modeling_tf_utils.py:2830] 2024-02-29 20:59:44,616 >> loading weights file microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract/pytorch_model.bin
[INFO|modeling_tf_pytorch_utils.py:184] 2024-02-29 20:59:45,290 >> Loading PyTorch weights from /root/autodl-tmp/BioREx/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract/pytorch_model.bin
[INFO|modeling_tf_pytorch_utils.py:187] 2024-02-29 20:59:45,855 >> PyTorch checkpoint contains 133,577,846 parameters
[INFO|modeling_tf_pytorch_utils.py:344] 2024-02-29 20:59:46,677 >> Loaded 0 parameters in the TF 2.0 model.

When I run the ‘scripts/run_biorex_exp.sh’ and predict using the model I got from my own training, it warns again. No parameters are loaded in and the result is really poor：

Loaded 0 parameters in the TF 2.0 model.

But when I use your pre-trained weights to predict directly it is normal.

loading weights file pretrained_model_biolinkbert/tf_model.h5
[WARNING|modeling_tf_utils.py:2966] 2024-02-29 20:55:57,942 >> All model checkpoint layers were used when initializing TFBertForSequenceClassification.

[WARNING|modeling_tf_utils.py:2975] 2024-02-29 20:55:57,942 >> All the layers of TFBertForSequenceClassification were initialized from the model checkpoint at pretrained_model_biolinkbert.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertForSequenceClassification for predictions without further training.