Arabic Social Media NLP at Scale

Scale, noise, and sparse metadata

The collection and preprocessing system handled more than 700 million raw social-media records containing repeated tweets and retweets, weak or missing location metadata, non-Arabic content, automated posting, spam, explicit-content links, noisy Unicode, informal spelling, and dialectal Arabic.

700M+raw social-media records handled

MSA + 5Arabic dialects covered by the wider NLP work

~10Kexpert-labeled samples for supervised modeling

An early location-filter experiment also exposed how sparse the platform metadata was: requiring known Jordan location metadata together with Arabic reduced a much larger raw stream to a small usable subset. Geolocation therefore combined multiple signals rather than relying on one field.

Arabic NLP and public opinion mining pipeline showing social data collection, MongoDB storage, filtering, Arabic preprocessing, FastText representations, supervised modeling, NLP tasks, and analytical outputs — **Pipeline overview.** Collection, filtering, Arabic preprocessing, representation learning, supervised modeling, and downstream analysis.

From raw stream to modeling corpus

Collection and preprocessing

Social streamsArabic tracking + location signals

→

Source filteringAutomation, spam and low-value sources

→

MongoDB corpusStable IDs, deduplication and thread structure

→

Arabic preprocessingUnicode normalization, cleanup and segmentation

→

NLP modelsEmbeddings + supervised classifiers

Filtering before modeling

Source metadata, language signals, geolocation evidence, and explicit-content/link filtering reduced automated and contaminated traffic before downstream analysis.

Database-scale deduplication

MongoDB aggregation, batching, and disk-backed execution replaced per-document Python cleanup when working sets reached hundreds of millions of records.

Arabic-specific normalization

Presentation forms, ligatures, letter variants, tatweel, diacritics, punctuation, URLs, mentions, and other social-text artifacts were normalized or removed before segmentation and representation learning.

Dialect-aware representation

The wider NLP work covered Modern Standard Arabic and five dialects. FastText, Word2Vec, and TF-IDF were evaluated in the 2017-2018 modeling stack, with subword representations particularly suited to noisy spelling and out-of-vocabulary forms.

Open collection, deduplication & preprocessing deep dive →

Modeling in the 2017-2018 context

The project predates transformer-based NLP becoming the practical default. Classical and neural approaches were compared across relevance, extremism-related categories, sentiment, emotion, and topic classification; FastText/Word2Vec and TF-IDF supported representation experiments, while LSTM-based models were selected for several downstream tasks after comparison.

NLP and modeling · 2017-2018

Filtered Arabic textCleaned, localized and deduplicated

→

Normalization + segmentationCanonical script and sentence structure

→

RepresentationsFastText, Word2Vec, TF-IDF

→

Supervised modelsClassical models + neural / LSTM experiments

→

Research outputsCategories, sentiment, emotion, topics and dashboards

Scope

This was collaborative research carried out in 2017-2018, and the analytical conclusions belonged to the wider project team. The collection, preprocessing, and NLP implementation described here is summarized from surviving project material without exposing raw social data, credentials, infrastructure addresses, or other private project material.