GIFCT’s Hash-Sharing Database

Hashes and the HSDB Explained

A hash is a numerical representation of digital content (such as a video, image, or PDF) that cannot be reverse-engineered to recreate the original material. There are different types of hashes, including cryptographic hashes and perceptual hashes, each serving distinct purposes. Cryptographic hashes (e.g., MD5, SHA-256) are used to identify exact matches, while perceptual hashes (e.g., PDQ, TMK) are designed to detect visually or structurally similar content.

The HSDB supports both types of hashing and allows member companies to contribute hashed content along with labels. These labels help describe the content, including its type, any associated terrorist entity, and relevant behavioral indicators. The process does not involve sharing user data.

Member companies regularly query the database to check whether content on their platforms matches existing hashes. The HSDB also includes a feedback mechanism, allowing members to indicate whether they agree with labels and whether the content meets inclusion criteria. This feedback helps refine future assessments across the network.

GIFCT enables this collaboration while respecting that members operate under different policies. It does not dictate how hashes should be used or how moderation should be enforced; instead, it facilitates information sharing to help reduce the spread of terrorist content online.

Hash-Sharing Database Taxonomy

Hashes added to the database must fit within GIFCT’s Taxonomy, otherwise understood as inclusion criteria. This taxonomy addresses content based on a terrorist or violent extremist entity producing the content and the type, or behavioral elements, of the content the entity produced.

This taxonomy contains a series of parameters for inclusion in the database that takes into account:

• the producers of the content being terrorist and violent extremist entities
• the types of terrorist or violent extremist behavior associated with the content and/or the offline violence the content depicts and/or relates to
• categorize the hashes shared in terms of the form of the content in terms of images, videos, PDFs, and URLs

Taxonomy Expansions

The current taxonomy of the database reflects several evolutions and expansions over time. The taxonomy originally addressed images and videos produced by entities on the United Nations Security Council’s Consolidated Sanctions List and the behavior of the content that entity had produced.

In 2019, the first expansion was made to include content produced by the perpetrators of an offline attack live streaming or recording their violence, related to GIFCT’s Incident Response Framework.

Most recently, in response to GIFCT’s 2021 Taxonomy Report comprising recommendations from global experts and our tech company members, the taxonomy has expanded to include terrorist and violent extremist attacker manifestos and URLs that direct people to where the content addressed in GIFCT’s taxonomy is hosted.

Expanding the taxonomy iteratively means GIFCT can empower our members to combat a wider range of terrorist activity, work to address the Islamist extremist-bias that currently exists in the larger counterterrorism field, and remain diligent to impacts on the human rights of those most vulnerable in this context.