BRANDEFENSE BRANDEFENSE
  • Home
  • Product
    How it works?
    Platform Overview
    Cyber Intelligence
    Brand & Reputation Protection
    Exposure Management
    Solutions
    Threat Intelligence Service
    Brand Protection
    Vulnerability Management
    Attack Surface Management
    Fraud Protection
    VIP Security
    Vulnerability Intelligence
    By Use Case
    Preventing Data Leakage
    Phishing Monitoring
    Account Takeover Detection
    Stolen Credit Cards
    Dark Web Monitoring
    Remediation and Takedown
    brandefense background
    Eliminate risks
    Explore the Brandefense
  • Blog
  • Resources
    Security News
    Threat Intelligence Researches
    Digital Risk Protection – FAQ
    We in the Press
  • Partners
    Channel Partners
    Deal Registration
  • Company
    About Us
    Career
    Privacy Policy
    Terms of Use
    Contact Us
Free Trial

BRANDEFENSE

  • Home
  • Product
    How it works?
    Platform Overview
    Cyber Intelligence
    Brand & Reputation Protection
    Exposure Management
    Solutions
    Threat Intelligence Service
    Brand Protection
    Vulnerability Management
    Attack Surface Management
    Fraud Protection
    VIP Security
    Vulnerability Intelligence
    By Use Case
    Preventing Data Leakage
    Phishing Monitoring
    Account Takeover Detection
    Stolen Credit Cards
    Dark Web Monitoring
    Remediation and Takedown
    brandefense background
    Eliminate risks
    Explore the Brandefense
  • Blog
  • Resources
    Security News
    Threat Intelligence Researches
    Digital Risk Protection – FAQ
    We in the Press
  • Partners
    Channel Partners
    Deal Registration
  • Company
    About Us
    Career
    Privacy Policy
    Terms of Use
    Contact Us
Top Open Source Intelligence (OSINT) Tools for Dark Web

Top Open Source Intelligence (OSINT) Tools for Dark Web

BRANDEFENSE
Dark Web
30/07/2022

Last updated on December 12th, 2022 at 01:40 pm

Table of Contents

  • Abstract
  • Introduction to the Dark Web
  • Why is Darkweb so important in terms of Cyber Threat Intelligence?
    • Why we choose the TOR Hidden Network
  • Dark Web Monitoring with OSINT
    • How to get onion links over TOR
      • Ahmia
      • Hidden Wiki
    • How to collect Data on These Links
      • torBot
    • How to Process This Data
  • Conclusion

Abstract

The dark web, an intentionally hidden part of the internet, helps protect internet users’ privacy from traffic analysis attacks. This portion of the internet can only be accessed through specialized dark web browsers or technologies.

Manual research, which analysts widely use, is both time-consuming and ultimately inefficient. Some studies have used automated mechanisms to discover DarkWeb, but information about studies that systematically investigate or evaluate the content contained in its hidden network is scarce. This article highlights technological challenges when exploring illegal and extremist content using tools that can shed light on this anonymous network.

OSINT, or Open Source Intelligence, can scan to Decipher and find connections between criminals and other actors around them. Specialists can also use it to monitor information such as possible attacks and internal threats, as well as to neutralize or disrupt plans in real-time.

Osint sources include newspapers, magazines, news, conferences, photographs, video, social media accounts, blogs, press conferences, and academic documents. With the spread of the Internet and the increase in the use of social media, there has been a significant increase in OSINT sources. The point to be considered is collecting information from resources that do not require contact with the target. The combined and automated use of OSINT techniques allows a threat actor to reveal its name, location, IP address, or image.

In the light of this information, we will try to convey a context to you in attempting to use OSINT techniques in a combined way in Darkweb research.

Introduction to the Dark Web

Deep web and darknet are two very confused and very different concepts. The term “Deep Web” denotes a class of Internet content that, for various technical reasons, is not indexed by search engines. Resources indexed by a search engine, whether or not a user to the site from the home page to ‘deeper’ to go, must have valid credentials such as user name and password. The part behind a page that is accessible by user verification in a media streaming service is considered part of the deep web because it is often blocked from accessing resources behind the home page of a search engine’s browser.

Sites where many authentication processes are required, such as email account contents, social media accounts, online banking accounts, and messaging application contents, constitute the deep web content. In addition, information such as legal documents, scientific research, academic studies, and databases of private institutions are also stored here.

The Dark Web is the World Wide Web Content contained in a private network: Overlay networks that use the Internet but require specific software, configurations, or authorization to access it. Through the dark web, personal computer networks can communicate and conduct business anonymously without revealing identifying information, such as the user’s location. This also makes the dark web a subset of the bars we have on the Internet—websites on the Dark Web work in their unique environment, separated from the sites of surface sites. Cybercriminals use the dark web to coordinate and carry out cyberattacks, sell stolen data and even exploit it to companies.

full

The dark network is often one of the issues that always maintain the mystery, with many misunderstandings. To take advantage of Darkweb, we need to analyze and correlate the data correctly. It can then become a valuable pool of information used in cyber security.

In this article, we will describe examples and scenarios through TOR, the most popular hidden network, when we talk about the dark web.

Why is Darkweb so important in terms of Cyber Threat Intelligence?

The Dark web can be a valuable source of threat intelligence, where analysts can learn about how cyber attacks are carried out, stolen data, which attack tools are for sale and purchased, and the success rates of current cyber attack campaigns.  But finding, this intelligence and creating a complete picture of the threat environment can be complex and requires a thorough understanding of the dark web and how to conduct an investigation.

Some threat information that analysts can find on the Dark Web include:

– Exposed Leaks, names, email addresses, and precision assets related to your organization are usually offered for sale in dark web markets.

– Vulnerabilities are where security vulnerabilities about popular software used in many companies are sold.

– Threat Campaign, the necessary data can be accessed to track a new cyber attack campaign related to different sectors/scenarios.

– Digital Asset Accessing is sold, such as the database, critical servers, and infrastructures provided to sell infrastructure.

Following the dark web from a cybersecurity perspective gives us in-depth information about exciting monetization methods for criminals. By following these techniques and tactics, the next attack can be predicted and creates an essential context for us for pre-attack measures.

For these reasons, the dark web is an essential resource for OSINT because it makes up a large part of the internet and has rich content. Moreover, with dark web data collection, organizations or states can make discoveries about data breaches and illegal activities and take various measures accordingly. Here we will explain how data can be extracted via the dark web using OSINT techniques, how it can be analyzed, and the sample tools at the correlation point.

Why we choose the TOR Hidden Network

Informally, the Dark Web refers to the small portion of the Deep Web (the part of the Web which is usually considered to be beyond reach from current search engines) based on darknets. Common darknets include, among other smaller P2P networks, FreeNet, the Invisible Internet Project (I2P), and Tor. In the case of Tor, Tor hidden services are used to access different applications such as chat, email, or websites through the Tor network.

In this article, we focus in particular on the analysis of web- sites hosted on Tor hidden services — due to Tor’s much more enormous popularity among users, which comprised around 7,000 relays or proxies by the time of this writing. The Tor network is based on the onion routing technique for network traffic anonymization. Due to its nature, Tor hidden services are used for an extensive range of (cyber)-criminals activities. Thereby, several studies focused on how to discover, access, crawl, and categorize the content of the Dark Web.

Dark Web Monitoring with OSINT

Although it is technically possible to create your dark web monitoring program, it is unrealistic. There are many platforms on the Dark web where personal information is sold or traded will require a high level of computational power and systematization, constantly scanning each of them. Knowing how to make the right business plan and solutions according to the problem is one of the challenging aspects of the job.

In this article, We will continue with a structure setup consisting of a simple methodology comprised of several steps and tools that you can successfully automate these steps.

These steps are mainly:

  • How to get onion links over TOR
  • How to search on these links
  • How to collect the data through these connections
  • How to process this data

Of course, the number of vehicles here can be increased, but we wanted to show you a few sample vehicles and for what purpose we use them to provide you with an overview from here. Over time, the tools may change, but the steps toward the goal will remain the same.

How to get onion links over TOR

Due to the TOR Network architecture, it is not easy to find relevant content because it is not suitable for the search engine structure used by central systems. Therefore, we try to solve this problem with some solutions close to the search engines we use in daily life.

As a first step, focus on deep-web search engines. The most popular are Ahmia.fi(http://ahmia.fi/)  and The Uncensored Hidden Wiki, these two work with a different logic from each other.

Ahmia

Juha Nurmi developed Ahmia as part of the Tor Project, which is one of the closest things to a deep web search engine. Ahmia essentially collects .onion URLs from the Tor network, then feeds these pages to their index, provided they don’t contain a robots.txt file saying not to index them. In addition, Ahmia allows onion service operators to register their URLs, enabling them to be found. Through continuously collecting .onion URLs, Ahmia has created one of the most extensive indexes of the deep web.

full

Hidden Wiki

Uncensored Hidden Wiki works slightly differently. Anyone can register with an uncensored hidden Wiki; after that, everyone can edit connections in the database. The search engine works by calling the descriptions of the pages given in these links. “.onion” since domain names are changed very often; crowdsourcing links is one of the best ways to collect many valuable URLs and keep them up to date.

full
Quick Tips

The Hunchly Dark Web mailing list provides daily reports of hidden services on the Tor network. Combined with the other two alternatives in this list that can be taken daily, it can provide actionable intelligence findings.

How to collect Data on These Links

torBot

torBot is a dark web OSINT tool. It is written in Python and is open source. Dark on the web .scans sites with the onion extension. Some features include:

  • Returns Page title and address with a short description about the site.
  • Save crawl info to JSON file
  • Crawl custom domains.
  • Checking if the link is live.

Due to these features, the torBot tool turns out to be a tool that performs many tasks on its own with all the solutions we need.

full

Information collection processes on the Dark web with OSINT can be easily performed using various tools. One of these tools, TorBot, is a valuable tool with ease of use and essential information that it gives as output. It is important to use similar tools to shorten the information collection process.

How to Process This Data

We manually collected Tor links and obtained other related links using some tools. After this step, which we performed in a semi-automated way, we saved it on accessing the contents of the pages and analyzing them. The point of obtaining real intelligence will now depend on the correct correlation and storage of this data and the analysis of it.

At this point, we can use Maltego to visualize the data. Then, you can interact with the Kibana interface by holding the data on Elasticsearch to strengthen the context by typing complex queries.

As we mentioned at the beginning of our article, the main complicated and troublesome part is setting up the environment to be analyzed after the data is collected using OSINT techniques.

full

In this article, we discussed the technical difficulties and measures taken to address large-scale web crawls specific to the Tor network. Analyzing big data posed significant challenges. It will often be more effective to get a dark web intelligence service instead of designing these systems.

Conclusion

Open-source intelligence provides information security decision-making as information about malicious activities potentially affecting the business. However, while open-source intelligence using surface internet is more encountered, We know that using the Darknet is less common for typical cyber security analysts.

The difficulties of using Darknet for open source intelligence require different solutions at many points due to the use of structures that are not in central architecture. Researchers can use many other tools for the time in their work on Darkweb. It may vary in the future in the tools and systems shared here. However, the intelligence-creating system will not have many changes.

It is more than a Tool promotion to be explained here, rather than awareness, to create a perspective on Darkweb intelligence activities.

The research has tried to emphasize how data collection should be done systematically and where attention should be paid. It will require serious time and focus.

Stay tuned to the point where we can get more information about the threat actors in the changing and developing world and provide more intelligence combined with OSINT techniques.

Ahmia DarkWeb Deep Web Hidden Wiki OSINT TOR torBot
Share on Facebook Share on Twitter
Search
Categories
APT GroupsBlogDark WebDRPSFraudRansomwareSector AnalysisSecurity NewsVIP SecurityWe in the PressWeekly Newsletter
Recent Posts
  • Perspective of the Month | APT Groups
    Perspective of the Month | APT Groups
  • BellaCiao: The New Malware From Iran’s Charming Kitten
    BellaCiao: The New Malware From Iran’s Charming Kitten
  • Security News Digest | Security Newsletter | April 27, 2023
    Security News Digest | Security Newsletter | April 27, 2023
  • Cyber Security Trends in 2023: What You Need to Know
    Cyber Security Trends in 2023: What You Need to Know
2023 Ransomware Trends Report
Let’s Dive in Ransomware Attack Trends
Report

Let’s Dive in Ransomware Attack Trends

Download Report
Follow us!

Continue Reading

Previous post

Security News – Week 30

security news – week 30
critical vulnerabilities affecting 1000+ organizations detected in filewave mdm solution
Next post

Critical Vulnerabilities Affecting 1000+ Organizations Detected in FileWave MDM Solution

particle element
We know what hackers know about you
Our cyber threat intelligence and security research team is ready to help you.
Request a demo
Free Trial
Contact
Login

Follow us on

brandefense logo brandefense

Brandefense is solving SOC’s complex challenges. We are here to help Brandefense customers to protect their brands and reputations against cyber threats.

United States:

300 Delaware Ave. Ste 210 #328 Wilmington, DE 19801 / USA

Turkey:

Üniversiteler Mahallesi, 1605.Cadde, Kapı No:3/1, No: 204, 06800 Çankaya/Ankara 06800

© 2022 Brandefense. All rights reserved.

Solutions
Threat IntelligenceBrand ProtectionVulnerability ManagementFraud ProtectionVIP SecurityAttack Surface ManagementVulnerability Intelligence
Use Case
Data LeakagePhishing MonitoringAccount Takeover DetectionStolen Credit CardsDark Web MonitoringRemediation / Takedown
Partners
Channel PartnersDeal Registration
Company
AboutCareerPrivacy PolicyTerms Of UseContact
Manage Cookie Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
Manage options Manage services Manage vendors Read more about these purposes
View preferences
{title} {title} {title}
Close
Search

Hit enter to search or ESC to close