Skip to main content

Navigating the GenAI Frontier: Transformers, GPT, and the Path to Accelerated Innovation ( LLM )

 


Historical Context: Seq2Seq Paper and NMT by Joint Learning to Align & Translate Paper :

* Seq2Seq model introduced in "Sequence to Sequence Learning with Neural Networks" paper by Sutskever et al., revolutionized NLP with end-to-end learning. For example, translating "Bonjour" to "Hello" without handcrafted features.

* NMT by "Joint Learning to Align and Translate" (Luong et al.) improved Seq2Seq with attention mechanisms. For instance, aligning "Bonjour" and "Hello" more accurately based on context.



Introduction to Transformers (Paper: Attention is all you need) :

* Transformers introduced in "Attention is All You Need" paper.

* Replaced RNN-based models with attention mechanisms, making them highly parallelizable and efficient.


Why transformers :

* Transformers capture long-range dependencies effectively.

* They use self-attention to process tokens in parallel and capture global context efficiently.



 Explain the working of each transformer component :

* Input Embeddings: Tokens are embedded into high-dimensional vectors.

* Positional Encoding: Adds positional information to embeddings.

* Encoder: Processes input through self-attention and feedforward layers.

* Decoder: Generates output based on encoder's representation and target sequence.

* Attention Mechanism: Computes weighted sums of input embeddings.

* Feedforward Neural Networks: Apply non-linear transformations to attention outputs.



How is GPT-1 trained from Scratch? (Take Reference from BERT and GPT-1 Paper) : 

* GPT-1 pre-trained using unsupervised learning on large text corpus.

* Includes Masked Language Modeling (MLM) to predict masked words and Next Sentence Prediction (NSP) to predict sentence relationships.

* Learns bidirectional context and semantic representations during pre-training.



@InnomaticsResearchLabs #InnomaticsResearchLabs 

Labels:

Comments

Post a Comment

Popular posts from this blog

Language Modeling ( LLM )

Language Modeling ( LLM ) *   What is Language Modeling :   Language modeling powers modern NLP by predicting words based on context, using statistical analysis of vast text data. It's essential for tasks like word prediction and speech development, driving innovation in understanding human language. Types of language models :  N-gram Models  : These models predict the next word based on the preceding n-1 words, where n is the order of the model. For example, a trigram model (n=3) predicts the next word using the two preceding words. N-gram Language Model Example : Consider a simple corpus of text: I love hiking in the mountains. The mountains are beautiful. Trigram Model: P("in" | "hiking", "love") = Count("hiking love in") / Count("hiking love") P("are" | "mountains", "the") = Count("mountains the are") / Count("mountains the") Neural Language Models: Neural network-based language...
Post a Comment
Read more

Machine Learning

    Machine Learning                                                                                                                                                                         - Arthur Samuel in 1959 AI-ML-DL Artificial  Intelligence:        * Making Machine to think analysis and Make a decision * Computer that can imitate human intellect and behavior EX: Self Driving cars, Robots Machine Learning:  * Machine Learning is a subset of AI that uses Statistical Learning algorithm to build system that have the ability to automa...
Post a Comment
Read more