Streaming collection
The Twitter collector used the streaming API with Arabic language filtering, a Jordan bounding box, and a broad Arabic tracking vocabulary. Incoming records were written directly to a MongoDB replica set; tweet IDs became document IDs so duplicate-key errors could be detected at ingestion, while reconnect and failover handling supported long-running collection jobs.
Automation, spam, and sparse metadata
More than half of an early raw tweet stream was identified as automated or spam-generated. The later collector encoded roughly 65 source patterns associated with auto-posting applications, link farms, utility/news integrations, and adult-content sources, allowing much of that traffic to be rejected before downstream analysis.
A separate Arabic explicit-content/profanity lexicon was used during preprocessing. URLs and mentions were removed, and adult-content/link spam was treated as data contamination rather than allowed to distort language models and classifiers.
Source-level filtering
Known automated posting tools and recurring spam sources were matched from tweet source metadata, avoiding text-level analysis where the source itself already identified low-value automation.
Geolocation
Platform geotags were sparse. Processing combined explicit country/place metadata with coordinate bounds and profile-location matching using Arabic and English spellings of Jordanian cities.
Language
Arabic and undetermined-language records were retained for deeper processing rather than relying on a single platform language flag.
Duplicate and thread structure
Retweets, quotes, and replies could repeat text while carrying useful graph/thread relationships, so duplicate removal preserved representative records and related-post structure.
MongoDB-scale deduplication
The cleaning notebooks treat deduplication as a database workload rather than a Python-loop cleanup. Records were processed in large batches and grouped by tweet ID with aggregation stages that kept one representative document, replaced the grouped root, and wrote deduplicated batches back with $out. Disk-backed aggregation (allowDiskUse=True) was used when working sets exceeded memory.
The surviving notebooks also retain earlier per-document lookup and batched-insert approaches, making the shift in execution strategy visible as the corpus grew.
Arabic script and social-text normalization
The preprocessing code maps Arabic presentation forms and character variants back to canonical base letters, then applies social-text cleanup before tokenization and representation learning.
Unicode canonicalization
Contextual Arabic presentation forms, ligatures, and letter variants were mapped to stable base characters so visually equivalent text did not fragment the vocabulary into different code points.
Arabic-specific cleanup
Tatweel, harakat/tashkeel, special Qur'anic marks, Arabic/Indic numerals, and selected non-Arabic script ranges were stripped or normalized; repeated punctuation, spaces, and repeated alef were collapsed.
Social-text cleanup
URLs, @mentions, simple emoticons, and hearts were removed. Hashtag content was retained while the marker itself was stripped.
Segmentation
Arabic and Western sentence punctuation was normalized into sentence boundaries, while commas and related punctuation became spaces.
A generic ISRI Arabic stemmer was tested and rejected as a default because it produced damaging reductions on informal and dialectal words; stemming remained a domain-sensitive preprocessing choice.
MSA and dialect variation
The broader NLP work covered Modern Standard Arabic together with five Arabic dialects, where spelling is less standardized and social text mixes colloquial forms with MSA, names, hashtags, and borrowed words.
For the 2017-2018 modeling stack, custom FastText embeddings were trained on social text together with broader Arabic sources. Word2Vec and TF-IDF representations were also evaluated in the wider experimentation.
Human labels and model iteration
Roughly ten thousand social interactions were labeled with domain specialists through a dedicated annotation workflow. Inconsistent labels were investigated, categories were revisited, and the training pipeline evolved alongside annotation quality.
Scope
This was collaborative research carried out in 2017-2018, and the analytical conclusions belonged to the wider project team. The implementation is summarized from surviving project material without publishing raw social data, credentials, infrastructure addresses, or other private project material.