Between February 2022 and September 2025, Bellingcat staff and volunteers collected, geolocated, and shared more than 2,500 incidents of civilian harm following Russia’s full-scale invasion of Ukraine. 

As part of this effort, Bellingcat tested a new machine learning model intended to rank Telegram social media posts on their likelihood of containing incidents of civilian harm. 

This novel methodology dramatically reduced the search and selection time required, freeing researchers to focus on verifying incidents of civilian harm – not just searching for them. 

This piece documents our methodology, ethical considerations and lessons learned in the hope that others researching similar topics can benefit from our work. 

Open source research into civilian harm is still a relatively new field and it presents many challenges – one of the biggest is organising and sorting through the huge volume of user generated content being produced to find what is relevant. 

Machine learning, a form of artificial intelligence that uses algorithms to identify patterns from large amounts of data and make predictions, can make this task more efficient.

With ongoing conflicts involving large amounts of civilian harm occurring in Sudan, and much of the Middle East, this guide aims to offer those covering these conflicts an example of how machine learning can be used to help find and sort incidents. You can also access the Code Notebook for our model here.

We defined “civilian harm” not just as civilian deaths or injuries resulting from armed conflict, but also the broader and delayed effects on civilians from mental trauma, loss of livelihood, displacement, destruction of infrastructure and more. This definition was informed by the Protection of Civilians book on civilian harm. 

Initial Telegram Dataset 

Each Telegram post containing civilian harm which had already been manually verified by researchers was used to build an initial dataset of confirmed cases of civilian harm, which data scientists call positive instances. We collected a total of 5,848 unique URLs for these Telegram posts. For our manual collection we reviewed posts on relevant Telegram channels, working through oldest to newest posts each day. Assuming that a given post made it to our geolocated incidents list, it meant the researcher who flagged it also looked at the posts that appeared before and after it on Telegram and did not flag those ones, so we selected the 10 posts surrounding the verified civilian harm post as our additional dataset of posts that did not contain civilian harm. After excluding any deleted or duplicate posts, we ended up with 48,545 non-civilian harm posts, our negative instances. 

The choice to overrepresent negative instances aims at better reflecting the real world and increasing data available for model training. 

We enriched each URL with metadata from the Telegram API, such as the time of publication, reactions or textual content. As some of these posts had been deleted, we completed the missing data points with previously preserved versions from our Auto Archiver database, only available for the positive instances.

Feature Engineering

Training a machine learning model requires numerical data, as these models compute a prediction score based on mathematical operations.

We built these by converting raw data from our initial dataset, such as keywords signalling potential civilian harm, into numerical scores (or “features”) that the model could interpret, with the aim of increasing the model’s ability to identify patterns. This process, known as feature engineering, can significantly improve model results because it allows data scientists to suggest explicit context knowledge. 

A full list of features we used to train the model can be found in the code notebook accompanying this piece. Many features were directly inspired by researchers’ input from their experiences manually screening cases of civilian harm by sorting through a set number of Telegram channels and inspecting each post individually.

Several of the features used were directly built from the metadata contained in each Telegram post including media_type, day_of_week; or binary ones: forwarded, edited and reply_to. 

Other features included engagement information: views, forwards, total_reactions, and even individual features for most used emojis including the reaction_crying_face to count emoji.

Converting Text to Numbers 

To embed the experience from the manual collection process, researchers put together a list of keywords both in Ukrainian and Russian that, to them, signalled posts likely to  show civilian harm. For instance, “Шахед” and “КАБ” translated to “Shahed” and “Guided aerial bomb” respectively. We created a numerical feature to count their frequency. 

In addition, we included several generic English-language keywords which meaningfully signalled potential civilian harm, such as “injured”, “school affected” and “hospital affected” that were only used for generating semantic similarity scores. 

A semantic similarity score is a calculation used to determine the proximity in meaning between different words and phrases. To get the semantic similarity between the post text and each of our keywords, we represented each in a list of numbers via a Sentence Transformer model, which converts words into numerical representations called vectors that a computer can understand. 

We then calculated the level of similarity between each vector using cosine similarity, one of the most popular methods for measuring similarity between two pieces of text.

Due to how embeddings work, this calculation results in a figure on a scale from -1 (no semantic proximity) to 1 (same meaning). For example, the words “hurt” and “injured” would have a high similarity score, while “residential” and “injured” would have a negative score as the words are not semantically similar. 

Finally, to enable the model to identify the relevance of each post to civilian harm in Ukraine, we used a multilingual text transformer from the BERT family of language models to represent the entire post’s text as a vector of 768 numerical values. This model can efficiently represent text from many languages in a way that captures meaning: the same sentence in different languages will generate similar embeddings, and trained machine learning models can detect patterns in the embeddings. 

It is important to note that for this initial prototype of a civilian harm detection model, we did not include any features derived from media content such as photos and videos, although that would be a logical next step in attempting to improve model performance.

Selecting, Training and Evaluating Models

With 54,393 rows of 893 numerical features each, we selected four machine learning algorithms to train our predictive models. 

We chose Logistic Regression as a baseline algorithm due to its simplicity. We also selected three other “best in class” models, Random Forest, XGBoost, and LightGBM. These choices centred on the interpretability of the models and their ability to work on tabular data of this size. For example, we avoided neural networks due to a lack of interpretability and because those models work best with a larger dataset. 

To genuinely assess the performance of the trained models, we split our dataset into three parts:  

• A training set – the data the models were trained on (60 percent of the full dataset’s rows)
• A validation set – used for an intermediary evaluation when tuning model parameters (20 percent of all rows)
• A test set – hidden for the final performance assessment, so the models were evaluated on unseen data (remaining 20 percent of rows)

We used a stratified split to divide the dataset instead of a random split. This method ensured the proportion of positive instances (i.e. confirmed cases of civilian harm) remained consistent across all three sets at about 11 percent.

To measure the performance of machine learning models, we ran them through the test set and measured the number of correct and incorrect predictions. Models output a likelihood between 0 and 1 that each Telegram post contains civilian harm, and we tried to find a cut-off threshold that leads to a good balance between flagging almost every post (0.1) or flagging very few (0.9). 

There are two main types of evaluation metrics to gauge a model’s prediction power. Recall asserts what fraction of positive instances (i.e. known civilian harm posts) were correctly flagged as such. Precision measures the fraction of posts flagged as civilian harm that are indeed civilian harm posts.

During the training phase, we tuned the models to maximise average precision (PR-AUC), a metric that summarises precision across all recall levels. While this method also accounts for precision, it prioritises recall, which is preferable for this use case as it steers model selection to reduce the number of civilian harm posts that are skipped. 

The following table sorts models from best to worst PR-AUC against a baseline of a coin-flip predictor. ROC-AUC and F1 are two other evaluation metrics included as sanity checks. Simply put, ROC-AUC measures the probability of ranking two instances, one negative and one positive, correctly; F1 balances precision and recall equally and its best cut-off threshold value.

From these results, we selected XGBoost as our final model as it had the best scores when compared across all metrics.

Interpreting the Model

Because these models are interpretable, we can understand which features are the most useful when predicting whether a post includes civilian harm. The above table shows the top 10 features that most strongly signal the XGBoost model to make a decision:

• semantic_keywords_similarity: the semantic proximity between the post text and manually selected keywords “casualties”, “damage” and “civilian harm”
• bert:  the model was able to discern meaning from the text with the same strength as some of the other features in this list – there are three cases of this in the top 10
• reaction_crying_face: reactions with crying face emojis on the post
• group_of_messages: whether a post contains multiple media files
• keywords_in_text: the number of custom Ukrainian or Russian keywords in the post

These results generally tally with what you might expect when selecting Telegram posts for instances of civilian harm, including that posts that generate a lot of emotional engagement and posts using keywords about civilian harm were among those most likely to contain content related to this topic. Not all models had the same top features as XGBoost. In fact, for the Random Forest model the most important feature was the number of crying face emojis present in a post, a soft pattern highlighted by researchers when this methodology was first imagined.

LLM Results and Comparison

Retroactively, we decided to run a sample of the same test dataset through different large language models (LLMs) to gauge their ability to make these same predictions. 

We aimed to include an LLM-generated score as an extra feature for our trained models, which would be captured as relevant if it correlated with the correct predictions. 

To start, we selected two local models, the 1B and 4B variants of Gemma 3 from Google DeepMind, and two cloud-hosted models, Gemini 2.5 flash and Gemini 3.5 flash. With this selection, we hoped to compare results across a wide range of models’ expected performance. 

We generated a 400-row stratified sample (preserving the same proportion of real civilian harm instances) from the test dataset used for the custom models. For each of the four LLM models, we ran two tests: one where only the Telegram post message was sent, and another including both the message and the engineered features (excluding the text embeddings, as the model had direct access to the text). In the prompt for each model, we asked for a score between 0 and 1. We then evaluated the results as we did for the custom models. 

The above table shows that LLMs can indeed extract value from the engineered features. All four LLMs surpassed the baseline Logistic Regression model in our tests, yet none of them performed better than the other custom-trained models, and XGBoost remained the one with the highest PR-AUC. 

Still, Gemini 2.5 Flash performed better than its newer version 3.5 and even achieved a slightly higher best F1 score than any other model. While this is a good result, for the flagging of civilian harm posts, the PR-AUC remains the crucial metric, as it captures the model’s ability to identify infrequent instances of civilian harm while minimising false positives.

Ethical Considerations

Introducing an instrument of automated decision-making into a process of detecting civilian harm brings inherent ethical questions. These include automation bias, or how humans tend to blindly place faith in machine-generated recommendations; algorithmic bias, or how the results of these models echo the same patterns present in the training data, including under- or over-representation of types of civilian harm. 

The decision to test an automated methodology for this particular project came from the fact that there were limited resources for both steps in the process – the detection of potential civilian harm and its actual verification. Historically, we built an enormous backlog of unverified incidents because a lot of time had to be spent on monitoring the most recent events so that potential evidence would be captured and preserved as soon as possible. 

The automation of this process also reduced the exposure of researchers to a significant amount of unpleasant and distressing visual and text content, reducing the burden of exposure to traumatic content. 

For this project, we tried to ameliorate the ethical challenges with a number of strategies including randomly flagging posts not captured by any model, monitoring which features models relied on to make decisions, and by doing historical comparisons of patterns in data. 

Additionally, as stated above, for this initial prototype of a civilian harm detection model we did not include any features derived from the media content itself. In the future, it would be a logical next step in attempting to improve the model performance, to include the media from the posts – but using AI to review actual media comes with additional ethical challenges such as model bias.

Because of the opaque ownership of many LLM companies and their generative nature, the use of LLMs for an extra feature presented additional ethical challenges including privacy and safety concerns considering the sensitive nature of the data. Our model did not rely on LLMs, though we retroactively ran a sample through it. 

How the Model Fits into the Bigger Picture 

After selecting this model, we created a user interface where researchers could view a list of Telegram posts sorted from most to least likely to contain indications of civilian harm. The user interface was designed for quick triage and integration, where a positive confirmation from researchers would instantly send the post to the Auto Archiver (Bellingcat’s tool for preserving digital content) and then transfer it to ATLOS (our internal collaborative verification platform). Bellingcat staff and volunteers could then manually verify incidents. Researcher input was constantly stored so that this data could be used to improve the model in the future. 

Preliminary feedback indicated that the AI model was useful. Not only were we able to reduce time and harm from scouring through dozens of war reporting Telegram channels, researchers also reported that the stream of new posts being added to the verification backlog were capturing real and diverse cases of civilian harm. 

We recognise this model has much room for improvement and is a work in progress. Even though it can illicit diverse civilian harm posts, further tests and improvements (such as improved feature engineering and continuous evaluation) are needed before it can confidently be deployed.

Despite the focus on civilian harm and Telegram (highly popular in Ukraine and Russia), this pipeline is generic and can be adapted to other conflict monitoring tasks. How easily this can be done does depend on how open the social media platform is and whether it is possible to scrape posts from it. Apart from that, it is easy to incorporate new features and data, and cheap to automatically retrain, test and deploy models as the system receives more human input.  

Looking forward, sorting through overwhelming amounts of data in a conflict will continue to be challenging. Hopefully, this methodology can help newsrooms, conflict monitoring organisations, and others find the balance between ethical considerations and resources in order to carry out open source investigations on civilian harm and human rights violations. 

---

Editor’s note: This article was updated on July 3, 2026, to include a line outlining that the model described is a work in progress.

Bellingcat is a non-profit and the ability to carry out our work is dependent on the kind support of individual donors. If you would like to support our work, you can do so here. You can also subscribe to our Patreon channel here. Subscribe to our Newsletter and follow us on Bluesky here, Instagram here, Reddit here and YouTube here.

The post How to Use AI to Help Find Civilian Harm appeared first on bellingcat.

In the tiny town of Krasnopillia in rural Ukraine, the stillness of the night is shattered by the whine of a Russian drone. Seconds later, a community hospital bursts into flames. Sparks and debris rain down across the skeletons of trees as the fire sends plumes of smoke into the pitch-black sky.

Dozens of people are evacuated, according to local media reports – but as rescuers respond, in what appears to be a double-tap strike, Russian forces hit a shelter where more than 20 patients are huddled, including some with limited mobility. 

The strike in March 2025 comes just hours after a larger regional hospital in the northeastern Sumy governorate is targeted, decimating the primary health facilities serving the small town of Krasnopillia, whose prewar population was around 7,700. Healthcare services for the town “practically ceased” in the wake of the strikes, Olena Pryima, a local school director, told Bellingcat in a phone interview. 

“[The Russians] destroy the infrastructure so that people do not have the opportunity to live and exist normally. You cannot consult a doctor, nothing,” she said. “And now these people who remain, God forbid, the ambulance will not go there, just because the security situation does not allow it.”

Her own school was among the many buildings destroyed in Russian strikes, and she says it has been impossible to rebuild amid the ongoing war. “We try to heat some accommodations, in spite of everything … especially since this winter is very difficult,” Pryima said. “But we are not talking about rebuilding at all now. We have hope; we are collecting some documents [such as testimonies and damage assessments], since this will end someday – and then we can rebuild something.”

For the past four years, Bellingcat has been documenting and verifying incidents such as these, chronicling the extensive damage to civilian life and infrastructure after the onset of Russia’s full invasion which began in February 2022.  

In over 2,500 cases of civilian harm that we have verified – the vast majority of which occurred on Ukrainian territory, although dozens also took place in Russia – more than 1,100 residential structures were hit. Hundreds of other civilian sites such as schools, playgrounds, fire stations, hospitals, churches, cultural centres, museums, businesses and farms have been impacted too. 

Our data – which includes cases that Bellingcat researchers were able to definitively geolocate using open source evidence, and does not reflect the full extent of civilian harm across Ukraine – pinpoints more than 300 attacks on schools or childcare facilities, 170 hits on healthcare or humanitarian sites, and four dozen incidents targeting food and related infrastructure. 

While many attacks were clustered around four main cities – Kharkiv, Donetsk, Kherson and Kyiv – we documented strikes across all areas of the country. Of the weapons that could be identified through available open source information, cluster munitions were used in more than 100 cases. 

Cluster munitions, which are banned in more than 100 countries (but not Russia or Ukraine), have killed more than 1,200 people since the war began, with Ukraine recording the highest number of annual casualties worldwide from these weapons in 2024 for the third consecutive year, according to the Landmine and Cluster Munition Monitor. 

Bellingcat and members of its volunteer community logged all verified incidents of civilian harm on an interactive TimeMap over a four-year period spanning February 2022 to December 2025. The map is no longer being updated, but it remains online as an archive (and can be seen below). 

An interactive map detailing incidents of civilian harm between February 2022 and December 2025.

Since Russia’s invasion four years ago, the civilian toll in Ukraine has been stark, with around 15,000 killed – including more than 750 children – and 40,600 injured, according to a January 2026 report by the Office of the United Nations High Commissioner for Human Rights. 

An analysis last year by Armed Conflict Location and Event Data (ACLED) found that Russia followed “a persistent pattern of targeting of populated areas … often indiscriminate, other times more deliberate”. 

ACLED’s data for the period of February 2022 to late January 2026 highlights thousands of residential strikes across Ukraine, along with more than 750 attacks on healthcare facilities, 1,200 on educational sites, and 2,400 on energy infrastructure. A February 2025 World Bank report says it will take more than US$500bn to rebuild Ukraine. 

These numbers tell only part of the story. While much global media attention has focused on the politics of the Russia-Ukraine war, or highlighted strikes on large urban centres, civilians in remote rural villages have suffered outsized impacts from the destruction of schools, hospitals and cultural institutions – the key threads tying their communities together.

In Verkhna Syrovatka, a small village in Sumy of around 3,800 people, images from the scene of shelling in May 2025 revealed a massive hole in the community’s blue-roofed cultural house. Inside the facility, which once served as a place for rehearsals, children’s classes and folk ensembles, photographs and trophies could be seen amid piles of splintered wood and cracked concrete.

The village’s only school was also impacted, with many of its windows blown out, forcing classes to move online. This devastation reflects a countrywide trend, as UNICEF reports that Ukrainian children are falling behind in core subjects such as reading, maths and science.

Incidents of civilian harm recorder by Bellingcat in Verkhna Syrovatka. Readers can click or tap the dots to learn more about each incident.

Further south, the village of Opytne in the Donetsk region is gradually being erased, amid a series of Russian attacks dating back more than a decade to the 2014 occupation of the Crimean Peninsula. 

The village has changed hands repeatedly in recent years. In December 2022, drone footage revealed large-scale destruction of its residential area, including a medical office, music school and church. According to media reports, perhaps only half a dozen residents remain out of more than 1,000 who lived in the village a decade ago.

Image left shows the village of Opytne in 2021, before Russia’s full invasion (Credit: Airbus/Google Earth Pro). Image right shows the village of Opytne in 2024 (Credit: Maxar/Google Earth Pro).

A couple of months later, in February 2023 in Dvorichna, a rural settlement in the Kharkiv region, Russian forces launched another double-tap strike: as first responders searched for survivors from an earlier attack on the village council building, several emergency vehicles were hit. 

Located just south of the Russian border, Dvorichna has been occupied on and off since 2022. As a result, the village, whose population was roughly 3,500 four years ago, is estimated to house only 80 residents today.

Across Ukraine, the catalogue of horrors is endless. In Pravdyne, a small village in the Kherson region, the prewar population of more than 1,000 people was reported to have dwindled to fewer than 200 by late 2022. Corpses showing signs of torture have been exhumed from garden beds; in one case, residents reportedly buried the bodies of Ukrainian soldiers under slabs of slate to prevent dogs from reaching them. 

Incidents of civilian harm recorder by Bellingcat in Pravdyne. Readers can click or tap the dots to learn more about each incident.

In Sumy Oblast, Russian drone and missile attacks have forced residents to flee homes they inhabited for half a century. In the village of Hroza in northeastern Ukraine, one-fifth of the population died in a single attack while attending the funeral of a soldier, according to local officials.

What may never be calculated are the impacts this brutal conflict will have on future generations.

Incidents of civilian harm recorder by Bellingcat in Hroza. Readers can click or tap the dots to learn more about each incident.

Back in Krasnopillia, the local school director, Pryima says residents have tried hard to stay in what she calls “the zone of resilience”, but it has been a struggle.

“It’s very scary to fall asleep, because you don’t know if you’ll wake up in the morning,” she said, noting that residents live in constant fear of the drones that fly overhead, keenly aware that a bomb may drop at any moment. 

For Ukrainian children, the effects have been especially dire.

“Those children, before the full-scale invasion, were carefree, cheerful – what children should be,” Pryima said. “Those children are no longer there.” 

---

Bellingcat is a non-profit and the ability to carry out our work is dependent on the kind support of individual donors. If you would like to support our work, you can do so here. You can also subscribe to our Patreon channel here. Subscribe to our Newsletter and follow us on Bluesky here, Instagram here, Reddit here and YouTube here.

The post How Russia’s War Has Devastated Civilian Life in Ukraine appeared first on bellingcat.

A Bellingcat investigation has identified remnants of an AGM-88 series missile inside a three-storey apartment complex that was hit in Catia La Mar during the US military raid on Venezuela on Jan. 3, 2026 that reportedly killed at least one civilian. 

According to the Venezuelan independent media outlet, El Pitazo, Rosa Gonzalez, 79, was killed in this airstrike in the city of Catia La Mar in La Guaira State, 30 km north of the capital Caracas. The attack reportedly left a second individual severely wounded.

Bellingcat asked the US Department of Defense to confirm our findings. However, they stated that “a Battle Damage Assessment is ongoing”. We also reached out to the Department of State but they did not respond to our questions at the time of publication.

The Jan. 3 US attack on Venezuela targeted multiple locations across the country, including military installations and areas within and around the capital, Caracas. US military helicopters, jets and drones provided cover for an assault force that landed at Fort Tiuna,  the largest military complex in Caracas; captured President Nicolas Maduro and First Lady Cilia Flores, and flew them out of the country.

About 75 people, including civilians, were killed in the operation, US officials familiar with the matter told the Washington Post. Among the military fatalities were 32 Cuban and 21 Venezuelan soldiers, according to various media reports.

According to El Pitazo, a second woman, Yohana Rodríguez Sierra, 45, was killed, and her daughter wounded, in other strikes at a communications station at Cerro El Volcán. Multiple residential houses were also reportedly destroyed in the nearby area of La Boyera.

The military operation in Caracas follows a series of attacks on alleged drug boats which have reportedly killed at least 114 in the Caribbean Sea and the Eastern Pacific Ocean.

Identifying the Munition

Bellingcat has found videos showing the aftermath of the Catia La Mar attack and remnants of the munition filmed at the location. The strike destroyed some exterior walls of one apartment complex and caused extensive damage to at least two apartments.

Bellingcat geolocated the apartment complex to an area in Catia La Mar about 30 km north of Caracas (10.592796, -67.037721) and approximately 500 m east of a targeted air defence storage inside a military facility.

One video also reveals remnants of a munition. According to an analysis of visual evidence by Bellingcat, the remnants appear to show an AGM-88 series missile.

Another remnant of the AGM-88 series missile appears in a video published by Euronews. This remnant is a BSU-60 tail fin, that according to an analyst note on the Open Source Munitions Portal (OSMP), is used exclusively with the AGM-88 series missile.

The AGM-88 HARM/AARGM series are American-produced air-to-surface missiles that are designed to hit ground-based radar-emitting targets, such as air defence systems like the Buk-M2E used by Venezuela’s military, with several of them destroyed during the US military raid.

“Venezuela does not operate the AGM-88 HARM. Its F-16 acquisition occurred in 1983, when the US wouldn’t release anti-radiation tech to the region,” Dr Andrei Serbin Pont, International Analyst and President of the Regional Coordinating Centre for Economic and Social Investigations, CRIES, told Bellingcat. Later Israeli upgrades added guided munitions/AAMs, not ARMs, he said.

According to TheWarZone, Venezuela did not receive any precision air-to-surface munitions, such as the AGM-88 series missiles. The Stockholm International Peace Research Institute (SIPRI) Arms Transfer Database does not report any transfers of AGM-88 missiles to Venezuela.

US Navy aircraft were photographed in the region with AGM-88E AARGM missiles in the weeks before the operation. Chairman of the Joint Chiefs of Staff, General Caine, stated that jets of this type, designed to suppress and destroy air defences, took part in the operation.

Air Defence Systems Targeted

The US struck several air defence systems across Venezuela as part of the operation, with Buk-M2E launchers being destroyed at  La Guaira Port, and the Higuerote and La Carlota airbases. Satellite imagery from Vantor shows that the area near a BuK-M2E storage building at Fort Guaicaipuro was also struck, but no air defence systems can clearly be seen.

According to satellite imagery, Buk-M2E launchers appear to have been stored at the military base approximately 500m from the residential building that was hit.

Bellingcat was not able to determine what caused the missile to strike the apartment building or if Buk-2ME launchers at Catia La Mar or a different system were the intended target. 

The AARGM variant is capable of having designated missile “impact zones” and “avoidance zones” programmed to determine where the missile can or can’t impact when used on missions, a feature added to “prevent collateral damage”.

Bellingcat asked the US Department of Defense if any weapons used in the operation transmitted a weapons impact assessment or other data that indicated they hit an unintended or civilian location. They said that a Battle Damage Assessment is ongoing.

---

Carlos Gonzales, Giancarlo Fiorella, Jake Godin, Trevor Ball and Youri van der Weide contributed to this report.

Bellingcat is a non-profit and the ability to carry out our work is dependent on the kind support of individual donors. If you would like to support our work, you can do so here. You can also subscribe to our Patreon channel here. Subscribe to our Newsletter and follow us on Twitter here and Mastodon here.

The post Inside the Strike: The US Munition That Hit a Residential Building in Venezuela appeared first on bellingcat.