Data Preperation

Visulize the data used for training Eng-Arb translation experiments.

from data_pipeline import (
    train_loader, test_pairs,
    english_word_to_index, english_index_to_word,
    arabic_word_to_index, arabic_index_to_word,
    tokenize, encode_source, PAD_IDX,
)

src_batch, dec_in_batch, dec_tgt_batch, src_lengths = next(iter(train_loader))

The data we will be using has the follwing for each load (src_batch, dec_in_batch, dec_tgt_batch, src_lengths): - serc_batch: The source sentence (English) with padding - dec_in_batch: The decoder input sentence (Arabic) with padding - dec_tgt_batch: The decoder target sentence (Arabic) with padding - src_lengths: The actual length of the source sentence (without padding)

Below we will visualize the data for a single batch.

# Here is the shape for each:
print(f"src_batch shape: {src_batch.shape}")
print(f"dec_in_batch shape: {dec_in_batch.shape}")
print(f"dec_tgt_batch shape: {dec_tgt_batch.shape}")
print(f"src_lengths shape: {src_lengths.shape}")

src_batch shape: torch.Size([32, 13])
dec_in_batch shape: torch.Size([32, 10])
dec_tgt_batch shape: torch.Size([32, 10])
src_lengths shape: torch.Size([32])

WORD_IND = 10

#print the first source sentence in the batch
print("First source sentence in the batch:")
print(src_batch[WORD_IND])
#print the first decoder input sentence in the batch
print("First decoder input sentence in the batch:")
print(dec_in_batch[WORD_IND])
#print the first decoder target sentence in the batch
print("First decoder target sentence in the batch:")
print(dec_tgt_batch[WORD_IND])

First source sentence in the batch:
tensor([107, 106, 378, 703,   2,   0,   0,   0,   0,   0,   0,   0,   0])
First decoder input sentence in the batch:
tensor([   1,  110,  195,  972,  862, 1513,    0,    0,    0,    0])
First decoder target sentence in the batch:
tensor([ 110,  195,  972,  862, 1513,    2,    0,    0,    0,    0])

# print the first source sentence in the batch as words
print("First source sentence in the batch as words:")
print([english_index_to_word[idx.item()]
      for idx in src_batch[WORD_IND] if idx.item() != PAD_IDX])
# print the first decoder input sentence in the batch as words
print("First decoder input sentence in the batch as words:")
print([arabic_index_to_word[idx.item()] for idx in dec_in_batch[WORD_IND]
       if idx.item() != PAD_IDX])
# print the first decoder target sentence in the batch as words
print("First decoder target sentence in the batch as words:")
print([arabic_index_to_word[idx.item()]
      for idx in dec_tgt_batch[WORD_IND] if idx.item() != PAD_IDX])

First source sentence in the batch as words:
['that', 'was', 'years', 'ago', '<eos>']
First decoder input sentence in the batch as words:
['<sos>', 'كان', 'ذلك', 'منذ', 'عدة', 'سنين']
First decoder target sentence in the batch as words:
['كان', 'ذلك', 'منذ', 'عدة', 'سنين', '<eos>']

#print 30 decoder input sentences in the batch as words
for i in range(30):
    print(f"Decoder input sentence {i} in the batch as words:")
    print([arabic_index_to_word[idx.item()]
          for idx in dec_in_batch[i] if idx.item() != PAD_IDX])

Decoder input sentence 0 in the batch as words:
['<sos>', 'إنها', 'صغيرة', 'جدا']
Decoder input sentence 1 in the batch as words:
['<sos>', '<unk>', 'لي', 'عن', 'رحلة', 'سفرك', 'من', 'فضلك']
Decoder input sentence 2 in the batch as words:
['<sos>', 'لا', 'مشكلة', 'مطلقا', '!']
Decoder input sentence 3 in the batch as words:
['<sos>', 'هذا', 'يؤلم']
Decoder input sentence 4 in the batch as words:
['<sos>', 'آمل', 'ألا', '<unk>', 'نفس', 'خطأ', 'توم']
Decoder input sentence 5 in the batch as words:
['<sos>', 'كيف', 'تعلم', '<unk>', 'توم', 'لهذا', '؟']
Decoder input sentence 6 in the batch as words:
['<sos>', 'نحن', 'لا', '<unk>', 'الإنتظار', 'للعشاء']
Decoder input sentence 7 in the batch as words:
['<sos>', 'إنه', 'يأكل']
Decoder input sentence 8 in the batch as words:
['<sos>', 'ما', 'هي', '<unk>', 'عملك', 'في', 'المكتب', '؟']
Decoder input sentence 9 in the batch as words:
['<sos>', 'ذهبت', 'أيضا']
Decoder input sentence 10 in the batch as words:
['<sos>', 'كان', 'ذلك', 'منذ', 'عدة', 'سنين']
Decoder input sentence 11 in the batch as words:
['<sos>', 'لا', 'تقلق', 'أنا', '<unk>']
Decoder input sentence 12 in the batch as words:
['<sos>', 'أحتاج', 'التحدث', 'معك', 'عن', 'أمر', '<unk>']
Decoder input sentence 13 in the batch as words:
['<sos>', 'أنا', 'لا', 'أظن', 'أنها', 'تشبه', 'أمها']
Decoder input sentence 14 in the batch as words:
['<sos>', 'عند', 'كل', 'الناس', '<unk>', 'من', '<unk>', 'اليوم']
Decoder input sentence 15 in the batch as words:
['<sos>', 'هل', 'من', '<unk>', 'يا', 'سيدي', '؟']
Decoder input sentence 16 in the batch as words:
['<sos>', 'الحب', 'أعمى']
Decoder input sentence 17 in the batch as words:
['<sos>', 'أصلحت', 'الدراجة', 'بالأمس']
Decoder input sentence 18 in the batch as words:
['<sos>', 'هو', 'موهوب', 'جدا']
Decoder input sentence 19 in the batch as words:
['<sos>', 'نصحته', 'ألا', 'يأكل', 'كثيرا']
Decoder input sentence 20 in the batch as words:
['<sos>', 'أخبرني', 'توم', '<unk>', 'أفعل', 'ذلك', 'أبدا']
Decoder input sentence 21 in the batch as words:
['<sos>', 'لم', 'أحصل', 'على', 'خيار']
Decoder input sentence 22 in the batch as words:
['<sos>', 'هل', 'نسيت', 'إقفال', 'الباب', 'الليلة', 'الماضية', '؟']
Decoder input sentence 23 in the batch as words:
['<sos>', 'يصعب', 'علي', 'قراءة', 'هذا', 'الكتاب']
Decoder input sentence 24 in the batch as words:
['<sos>', 'أنت', 'دائما', 'ما', '<unk>']
Decoder input sentence 25 in the batch as words:
['<sos>', 'لا', 'أعرف', 'كيف', 'لكني', '<unk>']
Decoder input sentence 26 in the batch as words:
['<sos>', 'من', 'أنت', '؟']
Decoder input sentence 27 in the batch as words:
['<sos>', '<unk>', 'توم', 'كثيرا', 'إن', 'عرف', 'أني', 'قلت', 'لك', 'هذا']
Decoder input sentence 28 in the batch as words:
['<sos>', 'هل', 'هذه', '<unk>', '؟']
Decoder input sentence 29 in the batch as words:
['<sos>', 'الذي', 'قلته', 'كان', 'يجب', 'أن', 'يقال']