Abstract:
Sentiment Analysis is an ongoing field of research in Natural Language Processing (NLP)
particularly aimed at analyzing subjective and textual information to extract judgmental or
behavioral knowledge for the computational treatment of opinions and sentiments of
individuals. In transfer learning a model is pre-trained on a large unsupervised dataset and then
fine-tuned on domain-specific downstream tasks. BERT is the first true-natured deep
bidirectional language model which reads the input from both sides of input to better
understand the context of a sentence by solely relying on the Attention mechanism. This study
presents a Twitter Modified BERT (TM-BERT) based upon Transformer architecture. It has
also developed a new Covid-19 Vaccination Sentiment Analysis Task (CV-SAT) and a
COVID-19 unsupervised pre-training dataset containing (70K) tweets. BERT achieved (0.70)
and (0.76) accuracy when fine-tuned on CV-SAT, whereas TM-BERT achieved (0.89), a
(19%) and (13%) accuracy over BERT. Another enhancement introduced is in terms of time
efficiency as BERT takes (64) hours of pre-training while TM-BERT takes only (17) hours and
still produces (19%) improvement even after pre-trained on four (4) times fewer data.
