Streaming collection
The Twitter collector used the streaming API with Arabic language filtering, a Jordan bounding box and a broad Arabic tracking vocabulary. Incoming records were written directly to a MongoDB replica set. Tweet IDs became document IDs, so duplicate-key errors could be detected explicitly at ingestion, while reconnect and failover handling kept long-running collection jobs alive.
Filtering a hostile raw stream
Automated content was not a small edge case. During the project, more than half of an early raw tweet stream was identified as automated or spam-generated. The later collector encoded roughly 65 source patterns associated with auto-posting applications, link farms, utility/news integrations and adult-content sources, and rejected them before storage or downstream analysis.
A separate Arabic explicit-content/profanity lexicon was used during preprocessing. URLs and mentions were removed, and adult-content/link spam was treated as data contamination rather than allowed to distort the language models and classifiers.
Source-level filtering
Known automated posting tools and recurring spam sources were matched from the tweet source metadata, avoiding expensive text analysis where the source itself already identified low-value automation.
Geolocation
Platform geotags were sparse. The processing code combined explicit country/place metadata with coordinate bounds and profile-location matching using Arabic and English spellings of Jordanian cities.
Language
Arabic and undetermined-language records were retained for deeper processing rather than trusting a single language flag, because social-media metadata was imperfect.
Duplicate and thread structure
Retweets, quotes and replies could repeat text while still carrying useful graph/thread relationships, so duplicate removal had to preserve representative records and related-post structure.
Deduplicating hundreds of millions of records
The MongoDB cleaning notebooks show the problem being treated as a database workload, not a Python-loop cleanup. Records were processed in large batches and grouped by tweet ID with aggregation stages that kept one representative document, replaced the grouped root and wrote deduplicated batches back with $out. Disk-backed aggregation (allowDiskUse=True) was used when the working set exceeded memory.
The notebooks also preserve slower trial approaches based on per-document lookups and batched inserts. At this scale, algorithm choice and database execution strategy mattered more than compact code.
Arabic normalization was an NLP problem of its own
Social-media Arabic could not be passed directly into a tokenizer. The preprocessing code contains a large normalization table that maps Arabic presentation forms and character variants back to canonical base letters, then applies multiple layers of social-text cleanup.
Unicode canonicalization
Contextual Arabic presentation forms, ligatures and letter variants were mapped to stable base characters so visually equivalent text did not fragment the vocabulary into different code points.
Arabic-specific cleanup
Tatweel, harakat/tashkeel, special Qur'anic marks, Arabic/Indic numerals and selected non-Arabic script ranges were stripped or normalized. Repeated punctuation, spaces and repeated alef were collapsed.
Social-text cleanup
URLs, @mentions, simple emoticons and hearts were removed. Hashtag content was retained while the marker itself was stripped, preserving useful lexical information.
Segmentation
Arabic and Western sentence punctuation was normalized into sentence boundaries, while commas and related punctuation became spaces. The result was a cleaner sentence-level corpus for tokenization and embedding training.
A generic ISRI Arabic stemmer was tested and explicitly rejected as a default because it produced damaging reductions on informal/dialectal words. The experiments instead treated stemming as a domain-sensitive decision, with exceptions and dictionary-aware alternatives considered.
Formal Arabic was not enough
The broader NLP work covered Modern Standard Arabic together with five Arabic dialects. That changes the preprocessing problem: spelling is less standardized, vocabulary shifts by region, and social text mixes colloquial forms with MSA, names, hashtags and borrowed words.
Custom FastText embeddings were trained on social text together with broader Arabic sources. Subword representations were particularly useful for noisy and out-of-vocabulary forms, while Word2Vec and TF-IDF representations were also evaluated in the wider experimentation.
Human labels and model iteration
Roughly ten thousand social interactions were labeled with domain specialists through a dedicated annotation workflow. The work was iterative: inconsistent labels were investigated rather than treated as model failure, categories were revisited, and the training pipeline evolved alongside the annotation quality.
Scope
This was collaborative research, and the analytical conclusions belonged to the wider project team. The collection, preprocessing and NLP code described here is summarized from the surviving project source tree without publishing raw social data, credentials, infrastructure addresses or other private project material.