We’ve got bad news!

Information spreads via Internet not only by means of big online media and social networks. There is a huge ramified ecosystem of “junk websites”. “News” shared by such “media” is often of very low quality or even complete fake. Still it reaches around 50 million of visitors per month. For comparison “Ukrainska Pravda” (popular news media) reaches around 15 million of visitors. Hundreds of thousands of people read “junk news” daily. We’ve used the machine algorithms to examine approximately a million of texts published on such “junk websites” within a year. Here’s what we have found out.

Published: November 28, 2018.
Authors: Anatoliy Bondarenko, Nadja Kelm, Roman Kulchynsky, Nadiia Romanenko, Yaroslava Tymoshchuk

Civil servants from Verkhovna Rada (Ukrainian Parliament) once decided to strengthen the parliament’s fence with an additional tool and ordered a “spiral safety barrier”. Ukrainian journalists dug that information up and created an ordinary item for daily newsfeed. The information was supposed to add some traffic and to entertain the readers a little bit. MPs, fence and barbed wire – the theme was potentially interesting.

“A Barrier Between the Authorities and the People”: How This News Was Spreading

Each ball is a news item. Violet balls are for news from manipulative websites, gray ones are for news from the websites not studied by us and for mainstream media.

On the 14th of August this news appeared on Livyy Bereg («Лівий Берег») website for the first time. Later it was republished – still in an ordinary news manner – by other mainstream Ukrainian media. But in a few hours this news began spreading uncontrolled: the fence in the backyard had turned into the barrier between Ukrainian people and the authorities.

There are many cases like these. Just take a look into your commuting fellow’s screen – what is she or he reading while riding a bus? There is a great chance that you’ll be terrified by the headlines of the news – a dense flow of manipulations, fakes, horrors and entertainment. Or, if you are strong enough mentally, try to have a conversation about politics with a taxi driver. In most of the cases you would hear a range of absolutely incredible statements which have little in common with the reality. Ask them – where did he or she got to know it from – the answer is simple like that: “I’ve read it from the Internet”. Objects of our investigation are Internet media that serve as news sources for the readers who do not care too much about the quality and the information’s proofs.

Who is in our scope?

During 2017 we’ve compiled a list of relatively little-known websites where we noticed anti-Ukrainian or pro-Russian propaganda. We have collected about 150 such websites – and it is not their full number, of course. These sites have become the subject of our thorough study. Some of them got closed during this year, some have changed their addresses. There are websites that duplicate each other.

Also we have excluded 36 websites from the original list - their level of manipulative news had been defined as “less than 25%” according to our algorithm. As a result we’ve got 80 websites which spread the manipulations online.

Nothing Personal – It’s Just Business

As some of the owners and PR specialists of such media state, the major mission of these websites is very simple – to make money by posting pre-paid materials to destroy somebody’s reputation. Or, as an option, they publish the accusations against some well-known and rich personalities and then take “news” away from their websites after being paid for this.

Such websites also generate information flow that brings traffic. And it is a mix of faked news, real news, slightly changed news and misinterpreted news as well as exotic recipes and celebrities’ tempting photos. There are the titles like “Streets are overloaded with dead bodies” used for the news item about poisoning of 6 dogs in Berdyansk.

The difference of such websites from classic “yellow press” i that there is no real editorial office with real correspondents who create original content, no real contacts are provided, and of course no standards are obeyed. As for the design – it is usually very simple one, often using the basic templates.

As we have already pointed, according to the information provided by our informants, these media earn from publishing and then removing pre-paid compromising materials. Victims of online blackmailing, occasionally googling their names, are supposed to get scared and to try to “clean up” such information. This method works well if the person accused is not very popular in Internet media – in this case all the negative news items hit the top lines of the search engine results.

A distinctive feature of such publications is a victim’s full name in the headlines, sometimes even with patronymic. SEO-optimizing specialists explain to “customers” that this way it would be googled better. In “normal” media headlines usually try to attract the attention with a victim’s work or family relations – for example, not “(little-known) Ivan Petrenko has got caught on a bribe”, but “Petro Kyrychenko’s (more famous one) father-or-mother-or-other-relative-in-law has got caught on a bribe!”

Compromising publications on such websites are targeting future “customers” who would come and offer money to move away the discrediting information. And this is the main source of their earnings.

While there are no such “clients” these websites imitate the work of “normal media” reposting various content with little or none modifications – sometimes this content is “normal” and sometimes not. They hunt for readers who would click the headline and increase the number of site’s visits. One visit is not an equivalent of a one person from the audience – please read here how the visits are being counted.

Unlike many top rated media, which though posting the pre-paid materials but rather positive or promotional ones, "junk websites" do not care about the quality of their content. It could be a completely faked story and a completely true one (with too little evidence to be published by “normal” media). Accusations could be based on rumors, Photoshop drawn “documents”, fake web-pages of foreign media or on Facebook posts, usually from the groups that have been created for these particular goals.

“The publisher” cannot be found or threatened – he and his servers are placed abroad. There is only a possibility to clean away discrediting information with a help of a special “negotiator”. But no one denies that it is a hook: if you pay once – you’ll be either paying regularly or will get the tones of discrediting materials in case of refusal. Still “experienced victims” of such informational attacks know: if a “client” does not succumb to blackmail, those “journalists-pirates” will ride away in search of new victims.

The price-list for such “deals” (there are few poetic names for them like “The Silent Promise” or “The Quiet Don Operation”) totally depends on how impudent the “website” is and on how strongly the “victim” of the publication is scared. As a rule the prices start from 1000 USD for a little-known site from the second dozen of Google results.

According to our informant, each customer who orders a trash campaign dreams big – that the news will make its way into the headlines of “normal” media or even more – on screens of top TV channels.

“One cannot just offer “smelly” topics to these sites directly”. But there are some other ways to do it – for example, faked accusation first gets published on a junk website, later it is posted by some “opinion leaders” on their Facebook pages.

The opinion leader can be a real one, famous or little-known, but also it can be a “bot” – a fictional person, who “all of a sudden” gets interested in a particular topic and becomes an “expert” in it. Thus a fake gets into the newsfeeds with references to “opinion leaders” and their social networks' pages. Very often such “news” is reposted by top ranking media due to laziness or inattention of the editor (night or weekend editor usually). “Another way to attract the readers of top ranking media and to make them read these faked news is to place a banner” – states our source.

From an Unknown Facebook Account – up to “According to News Reports”

1 “A piece of news” is published by an unknown Facebook account.

2 Accusations appeared at unknown Facebook account are republished by “decent media” with references to a “judge who is hiding under this FB nickname”.

3 “The news” is legalized. Voila! Now the information from an unknown Facebook account has turned into “according to news reports”.

From an Unknown Facebook Account – up to “According to News Reports”

1 “A piece of news” is published by an unknown Facebook account.

2 Accusations appeared at unknown Facebook account are republished by “decent media” with references to a “judge who is hiding under this FB nickname”.

3 “The news” is legalized. Voila! Now the information from an unknown Facebook account has turned into “according to news reports”.

“Opinion leader” who spreads faked or manipulative news from a junk website does not necessarily do it for money – he or she can be just not smart enough to recognize news and standards or to be truly indignant by the “news”. But in most of the cases Facebook “leaders” do know what they post and share.

“As in beauty industry or at FMCG market, where bloggers do not conceal that they take money for commercial posts, there are purchasable opinion leaders on the market of information wars as well” – our source says.

Such an “expert” can get paid only once or be paid regularly. He can repeat the information published by a low quality website thus providing some legitimacy for publishing the fake in a top ranking media newsfeed.

One can just call to some “newsrooms” and offer the materials for publishing. For example we called newsonline24.com.ua (21 000 visits per month in October 2018 according to SimilarWeb) and they had asked for just $65 for a faked news discrediting one of potential presidential candidates. However there are players who work only with well-tried PR agencies.

In some media an owner, an editor and a journalist is the same person. In others – an owner can have a number of such media. They do not generate their own content besides prepaid discrediting materials. In other cases they create content by multiple reposts – sometimes with original headlines, sometimes with rewritten “horrifying” ones.

These sites are not picky when they receive money for prepaid materials from “clients”. At some moment Russian propaganda can take advantage of such opportunities. We assume that these sites may accept large orders from PR agencies which in turn have contracts with Russian propagandists. But we have not caught anyone of them on spot. Instead we have been observing how junk websites repost Kremlin major theses.

Emotional Manipulations

Coloring and placing in this ranking indicate the percentage of emotionally manipulative news: 20-85%. The length of each rectangle means a number of visits per month (logarithmic scale with the base 2)

Scroll down to change the type of ranking

Place the mouse cursor on to see the connections

Emotional Manipulations

These are publications that contain insult or overuse of expressive means or obscene vocabulary. This “news” does not inform but influences the readers emotionally.

The classifier for Russian language correctly defines emotional manipulations in 85% of the cases. The algorithm recognizes 61 out of 100 emotionally biased news – the overall accuracy is higher, because there is a less amount of manipulative news. Out of 100 news which do not contain emotional manipulations the algorithm recognizes correctly 92 (and 8 are incorrectly marked).

The algorithm for Ukrainian language correctly classifies 88% of the news. The algorithm recognizes 40 out of 100 emotionally biased news and marks mistakenly 5 out of 100 with no emotional manipulations.

In this ranking listed are only the websites which contain over 200 of news items about Ukraine with more than 25% of manipulative content according to our algorithm.

The Manipulations with Arguments

The manipulations with arguments include logical errors in arguments, ill-founded allegations, appeals for “well-known” facts, one-sided coverage of the topic or speculations made by doubtful experts.

The algorithm for Russian language that classifies the manipulations with arguments defines 55 out of 100news containing such manipulations. This classifier is wrong in 6 cases out of 100 news without manipulations. In total the algorithm can identify correctly the manipulating with arguments in 82% of the cases.

The classifier for Ukrainian language can recognize 30 out of 100 news which contain manipulations with arguments. It marks falsely 4 out of 100 news, in which no manipulations are present. In total the algorithm classifies correctly the manipulations with arguments in 88% of the cases.

Total Score of Manipulativeness

It is the proportion of website materials in which the algorithm has classified at least one of the manipulations. Here you can see the ranking in spreadsheet.

Here you can see the ranking in spreadsheet.

Russian Propaganda

22 websites out of our range are the news resources from ORDLO (Temporarily occupied and uncontrolled territories of Ukraine) or Russian “trashy” propagandist media targeting Ukraine and its agenda. Out of all the news about Ukraine published on these sites 37 to 75% manipulate with emotions or arguments. For all that in Ukraine they do have 22.5 million of visits per month.

Total audience of low quality small websites that spread Russian propaganda is equivalent to the one of big Ukrainian media. For example “Russkaya Vesna” (Literally – Russian Spring) demonstrates 11.8mln of visits from the territory of Ukraine in October 2018. Compared to it are the numbers of “Ukrainska Pravda” – 16mln and “Novoye Vremya” – 10.25mln of visits per month for the same period. Still it’s possible that “RussPropagandos” simply buy this traffic just as their colleagues from Baltic States do. We have not investigated this particular issue.

Here are some of them: Politnavigator, Russkaya Vesna, Komitet Vyzvolennya Ukrainy (literally – The Committee for the Liberation of Ukraine), Ukraina.ru, Naspravdi.info. What they publish seems like a parallel universe. Their mission is to keep mentally stimulating their loyal followers so keep them “safe from the destructive reality”. “The plague of dehumanization devours Ukraine piece by piece” – this is an original text, the beginning of one of the publications on Politnavigator.

Ranking of Manipulative Websites

We calculated a share of manipulative news at each website. Below is the list of websites where our algorithm classified more than 25% of items as manipulative.

Ranking by type of manipulation:

20% - 85%, visits per month:
log2 scale

Total score of manipulativeness is the proportion of website materials in which the algorithm has classified at least one of the manipulations.

Emotional Manipulations

These are publications that contain insult or overuse of expressive means or obscene vocabulary. This “news” does not inform but influences the readers emotionally.

The classifier for Russian language correctly defines emotional manipulations in 85% of the cases. The algorithm recognizes 61 out of 100 emotionally biased news – the overall accuracy is higher, because there is a less amount of manipulative news. Out of 100 news which do not contain emotional manipulations the algorithm recognizes correctly 92 (and 8 are incorrectly marked).

The algorithm for Ukrainian language correctly classifies 88% of the news. The algorithm recognizes 40 out of 100 emotionally biased news and marks mistakenly 5 out of 100 with no emotional manipulations.

In this ranking listed are only the websites which contain over 200 of news items about Ukraine with more than 25% of manipulative content according to our algorithm.

The Manipulations with Arguments

The manipulations with arguments include logical errors in arguments, ill-founded allegations, appeals for “well-known” facts, one-sided coverage of the topic or speculations made by doubtful experts.

The algorithm for Russian language that classifies the manipulations with arguments defines 55 out of 100news containing such manipulations. This classifier is wrong in 6 cases out of 100 news without manipulations. In total the algorithm can identify correctly the manipulating with arguments in 82% of the cases.

The classifier for Ukrainian language can recognize 30 out of 100 news which contain manipulations with arguments. It marks falsely 4 out of 100 news, in which no manipulations are present. In total the algorithm classifies correctly the manipulations with arguments in 88% of the cases.

Here you can see the ranking in spreadsheet.

Russian Propaganda

22 websites out of our range are the news resources from ORDLO (Temporarily occupied and uncontrolled territories of Ukraine) or Russian “trashy” propagandist media targeting Ukraine and its agenda. Out of all the news about Ukraine published on these sites 37 to 75% manipulate with emotions or arguments. For all that in Ukraine they do have 22.5 million of visits per month.

Total audience of low quality small websites that spread Russian propaganda is equivalent to the one of big Ukrainian media. For example “Russkaya Vesna” (Literally – Russian Spring) demonstrates 11.8mln of visits from the territory of Ukraine in October 2018. Compared to it are the numbers of “Ukrainska Pravda” – 16mln and “Novoye Vremya” – 10.25mln of visits per month for the same period. Still it’s possible that “RussPropagandos” simply buy this traffic just as their colleagues from Baltic States do. We have not investigated this particular issue.

Here are some of them: Politnavigator, Russkaya Vesna, Komitet Vyzvolennya Ukrainy (literally – The Committee for the Liberation of Ukraine), Ukraina.ru, Naspravdi.info. What they publish seems like a parallel universe. Their mission is to keep mentally stimulating their loyal followers so keep them “safe from the destructive reality”. “The plague of dehumanization devours Ukraine piece by piece” – this is an original text, the beginning of one of the publications on Politnavigator.

How It Influences Media Field

We assume that a manipulative news business model exists because of the phenomenon which could be called “I’ve read it on the Internet”. What does it mean? Some readers see Internet as big, all-in-one mass media without differentiating various parts such as online platforms. These parts may vary by their mission, standards, values and reputation. But for many readers it does not matter where they have read this or that news on – Liga.net (real media with thorough fact-checking policy) or Skelet.info (junk website publishing trash). They believe in what they read if it affects them emotionally. However this hypothesis needs further research.

As it’s been already mentioned there is a number of ways to “legitimize” fakes and sometimes there are situations when unproved information from the very bottom of Ukrainian Internet gets into the news feeds of leading media.

All this together with political dependence of a large part of Ukrainian media and wide presence of Russian and pro-Russian Internet media make the informational space of Ukrainian Internet very susceptible to manipulations of public opinion.

According to IMI (Institute of Mass Information) research out of 50 most popular online media in Ukraine only about 20-25 do try to filter and fact-check their content in some way. But working in extremely polluted informational space causes even the most fastidious media to be at a higher risk of sharing false or manipulative news. And sometimes it happens. Nor should we exclude a deliberate work of Russian special services and propagandists when they launch their own messages into an informational space – both into the Internet in general and into leading media websites.

In a situation like this the mainstream media should lift up the “admission line” for the news to be published. The level to what it should be lifted taking into account the costs of labor and estimated profits – this is the subject of another discussion. What we can state right now is that the refusal to publish the quotes and comments of “political experts” or “opinion leaders” known for promoting prepaid theses will significantly refresh the whole informational atmosphere. “Ban” should also be applied for making news out of Facebook posts with exceptions for verified accounts only.

Manipuland: Key Topics of the Most Emotional News

Key topics of headlines of top 50 000 most emotional news (according to algorithm). Translated with Google Translate.

You can enlarge or tag the map

Out of all the headlines from the websites included into our study we’ve created a “news map” - to give an idea what these sites write about. It is a nice piece demonstrating what is published on the news websites daily. As an example we’ve used just one category of bad quality news – those that manipulate the emotions. Not all of them are anti-Ukrainian although such news makes a majority. With a help of the algorithm we divided 50 000 manipulative news collected within one year of study into cluster groups according to their headlines.

Then we named the biggest groups – picking up the names in accordance with a topic (the names are shown in medium font, for example – “Tymoshenko”) and after we united those groups with corresponding themes into 6 big topics – “About Ukraine”, “Authority/Politicians”, “War” etc. These 6 topics are highlighted by a larger font on the map. All the headlines belonging to one of these 6 major topics are marked by different colors.

After all in order to make the results to fit the screen we have left a random sample of 1500 news (they are divided proportionally between 6 major topic groups).

The most of the content (more than a half of all messages out of this sample of 50 000) is dedicated to Ukraine as a state. The most favorite topic is the “good old song of Russian propaganda”: “state-the-loser”, “state that does not exist”, “failed state”. The things are dreadful “on Ukraine” (Russian propagandists use the preposition ON instead of IN – as if Ukraine was an island), catastrophes and disasters happen every day or are just about to happen, all kinds of troubles exist inside the country and surround it from the outside. Any newsbreak is used to create “news” with “proper” messages and narratives. It is an important element of information war: Kremlin is using these narratives to spread apathy, helplessness, inability to make any decisions and actions among Ukrainian citizens.

As for the political figures appearing in this news the biggest portion of negative content is dedicated to current President of Ukraine Petro Poroshenko. According to the headlines analyzed President Poroshenko is a very, very, very bad man and a terrible politician: “an alcoholic”, “a thief”, “bold-faced liar” and so on. In sum both Ukrainian and Russian “junk” websites keep on pouring a steady stream of trash and negativity on Ukrainian President.

Also a lot of content, PR and propaganda was invested into Mikheil Saakhashvili, former President of Georgia. Though the newsbreaks for him existed in Ukrainian informational space only for a particular period of time, his name has turned out to be on the second place among the politicians according to a number of mentions in media. There is some negative news, but mostly it is either neutral or positive, often in the context of his opposition to President Poroshenko.

Those are just few examples. To know more about the content and narratives of the “news” – please see the map.

How to Counteract Manipulations: Brief Review of European Practices

In order to understand how to counteract or neutralize such informational influences we have looked at European experience. In most of the EU countries the state protects juridically the rights of specific individuals who have been directly targeted by the false information in media. There are also professional associations that monitor how standards and ethics in journalism are obeyed.

Two countries are implementing practices that could be interesting for Ukraine.

Spain. To detect and to counteract the disinformation is the function of National Cybersecurity Council. Spanish citizens changed the governmental branch responsible for fighting the hostilely intended disinformation – from a state regulatory agency to a security force department.

In summer 2018 lower house of Parliament of France adopted a law according to which a candidate participating in the election campaign has a right to go to law complaining about the media's manipulations with facts or information. If the court decides in favor of the plaintiff, the publication is to be removed immediately. And the trial itself in such cases should be completed within 48 hours. The critics of the law worried about the freedom of press. And during the debates in parliament there were doubts about its efficiency. The Senate passed the law from the second try. French journalists now actively discuss another initiative: a suggestion to introduce the certificates of true information. If professional community is able to work out and agree upon the standards of the document then whole media, including journalists and bloggers, would be able to receive and to place such certificates in their materials.

The Conclusion for Ukraine

The attempts of the state to regulate media in Ukraine can be very complicated. It is not always possible to find a real owner or an editor of the website. Also, and this is particularly the case in Ukrainian reality, state’s intervention could really be a threat to a freedom of speech.

State mechanisms of control in such a sensitive sphere as media can work efficiently only in the democratically developed societies with high level of trust to state institutions.

For example, when Malaysia and Egypt adopted their laws for fighting fake news the opposition parties in these countries and international human rights organizations had gone out on protests. They claimed it as an impudent censorship but not the laws to counteract fake information. In Malaysia the law had been canceled right after the new political force, more democratic one came into power.

French initiatives are being widely discussed in Ukraine now, but from our point of view we should rather study Spain’s experience more closely where the disinformation campaigns are detected by a state cyber security force, not a media regulator. When introducing such principles there should be special protective mechanisms provided to prevent the state agency from overusing its functions and turning into a censor. It is important for the organization counteracting disinformation campaigns to focus on the propaganda that threatens national security.

While developing the legislation on counteracting disinformation it is advisable to follow the rule “track where the money comes from” meaning that media should declare their source of financing.

But the development and implementation of any state policy takes a lot of time. And Ukraine is facing an election season that impedes decision making and policy implementation.

That is why the key role here should be played by civil initiatives and by journalists who do care about professional standards. At present the most efficient ways for Ukraine to defeat the detrimental effect of manipulative news are:

  1. Outreach campaigns explaining which news and media deserve to be trusted and which do not. It needs to be explained how to distinguish a piece of news that contains manipulations.
  2. Increasing the number of monitoring tools and organizations watching the journalistic standards being obeyed by all media.
  3. Solidarity and self-organization of key media, development of general principles and standards which are doable and acceptable for Ukrainian reality. In our opinion an obligatory component of such standards should be a strict ban on making the news out of different statements and announcements made by all sorts of “political experts” and “opinion leaders” previously known for their “purchasable” opinions.

There are currently no associations that unite leading Ukrainian media.

If we want media to be free of state regulation, which could easily turn into censorship, then we have to take the responsibility of working out the standards and monitoring them by our professional community. Either these are journalists who control themselves or are controlled by state, or it is a situation of anarchy and a society without normal news. What we are actually observing nowadays.

Reader’s Manual: How You Can Distinguish a Manipulative Piece of News

Here we suggest a check list, following which you can check the news and define for yourself whether this piece of news can be trusted or it should be omitted

Regular news is the facts. It is written briefly and it sounds rather boring in most of the times. If you are amazed, intrigued and still cannot find this news in the trustworthy media with normal editorial policy/mission – please doubt the news and check it twice.

Always look for references to original sources – they should be provided. A reliable source could be either well-known media with excellent reputation or official website of governmental body.

Professional design of the site, commercial banners within the common sense, the names of contributing authors and editorial contacts are the good signs.

A piece of news that is made out of blog or social media post is not the news in 99% of the cases.

Please pay attention to the recommendations “read also” inside the text: they often contain the judgments and make an associative series with the news. Sometimes the manipulation is hidden right here.

Usually in the articles authors have a slightly more freedom. But still the level of irony and sarcasm should not go beyond. Cursing, obscene language, and insults are not allowed even if you do agree that such “assessment” is deserved. All of these are the signs of manipulative texts.

Any time you notice an exaggeration or emotional expression – always ask yourself “what for”. Is it because the author wants to make the text more interesting for you to read or because they try to impose a certain opinion?

The news about some politician and his/her pre-election statements, which is published without any context, criticism or newsbreak, in most of the cases is pre-paid by this politician.

Be careful with experts – especially the political experts.

If the arguments in the analysis are unclear and confusing it does not mean that you do not understand something – it means that the arguments are either weak or wrong.

How the Study Was Conducted: Methodology

Please find more details about the technical part of the work here.

At the beginning of our study we invited the editors who work with the news on a daily basis and with their help we defined and described the most common types of manipulative news: “A complete fake”, “A manipulative headline / clickbait”, “A conspiracy theory / Pseudoscience”, “Emotionally biased news”, “Wrong arguments”, “Political conspiracies”, “A normal text”.

Then we downloaded around 7 thousand news from junk websites and from well-known websites with good reputation. After that we marked each piece of news – either as “normal” or as “corresponding to one of the manipulative types”. We studied only the news about Ukraine, omitting the themes of entertainment, sports, celebrities etc.

After that we trained the deep learning algorithm to define the types of manipulations in the news on the basis of previously classified news. While conducting our work we’ve found out that the most common types of manipulations are “emotional bias” and “wrong arguments”. Thus they were used to train the algorithm to “catch” the manipulations.

Please find the description of each category here.

We took the entire news flow from selected websites starting from December 2017 till October 2018 and calculated the percentage of manipulative news on each of them.

We did not do the fact checking of the news – that means that we looked not for fakes, but precisely for manipulations. Pure fakes (lies) in the news we marked as the “wrong arguments” type of manipulations. While marking the news “by hand” we have noticed that complete fakes are not very common.

We’ve ignored images and video for technical reason - though they could manipulate as well. We have not managed to train the machine to comprehend the text just as good as experts do. Algorithm discerned 60% of emotionally biased news in Russian and 40% of such news in Ukrainian. It discerned 55% of “wrong arguments” news in Russian and 30% of them in Ukrainian. While defining the manipulative news out of normal ones the algorithm was wrong in less than 8% of the cases. On big massive of data such percentage of errors allows to make a relative ranking of websites.

After classifier predicted manipulations in each piece of news, we summed up the results by calculating the percentage of manipulative news for each website.