Threat Intelligence Evaluation Checklist.

Print this page or tick it on screen. Run it once per provider during a trial, using your own company, sector and tools. Add up the ticks at the end and note every ‘partly’ as a follow-up question.

If you are still deciding the type of intelligence to buy, read the buyer’s guide to feeds, platforms and services first.

The MITRE ATT&CK knowledge base is a common shared language for attacker methods, so a provider that maps to it is easier to use with other tools.

A fair trial gives every provider the same inputs and the same time. Give each one your company name, main domains, sector and a list of technologies. Ask each to show what it finds about you, and compare the results side by side.

• Use the same period for all providers, long enough to include new events.
• Have two analysts rate items independently and compare their notes.
• Record which items led to an action and which were dismissed.
• Test the API and the integration with your own tools, not a demo environment.
• Ask one hard question and see how long the answer takes.

At the end, write one page with scores, open questions and a recommendation. This is also the document you will need when finance asks why this provider was chosen.

• Large numbers of sources with no names or types.
• Scores with no reasoning.
• Reports that could be about any company.
• No way to report false positives.
• Data that cannot be exported.
• Answers that avoid saying what is not covered.

None of these alone ends the evaluation, but several together suggest that the work will fall on your team.

Short definitions of the words used above, so that everyone in the buying team reads them the same way.

• Source type. The kind of place data comes from, such as open sources, criminal forums or sensors.
• Coverage. What part of the threat picture a provider can see.
• Enrichment. Context added to a raw item, such as the actor and the method.
• Confidence. How sure the provider is, and why.
• Indicator of compromise. A trace that shows a system may have been attacked.
• False positive. An alert that is not a real threat.
• API. A way for tools to exchange data automatically.
• STIX and TAXII. Common formats and a protocol for sharing threat information.
• Pivot. Moving from one item to related ones, such as from an address to a campaign.
• Attribution. Naming who is behind an attack, which should always be stated with its source.
• Trial. A time-limited test on your own company.

Prepare a short profile of your company: names, domains, sector, countries and the technologies you run. List the tools that must receive data and the person who owns each. Choose two or three recent events that affected your sector and ask each provider what it knew about them and when. Decide beforehand how many analyst hours the trial may use.

Agree who scores which section, so that the analyst judges context and noise, the engineer judges the API and the lead judges data rights and contract. Fix the scale in advance, for example yes, partly or no, and write one line of evidence for each score. Keep the sheet in one shared file so the results can be compared directly after the trial.

Finally, decide what result would make you stop the trial early, such as no items relevant to your company after a fair period. Writing this down protects the team from sunk-cost thinking and keeps the process fair to every provider.

The NIST Cybersecurity Framework lists cyber threat intelligence as an input to risk assessment (ID.RA-02) and to analyzing adverse events (DE.AE-07), and ENISA and CISA publish guidance on threat landscapes and sharing. Use them to check the provider’s claims against what independent bodies describe. For the same questions in RFP form, see the RFP question list. For a view of the provider’s product, see Brandefense cyber threat intelligence and apply the same sheet to it.

These public bodies are the references behind the criteria in this guide. Open them and cite the exact page you rely on in your own paper.

• NIST Cybersecurity Framework: the six functions and the outcomes under each.
• ENISA: European threat landscape reports and guidance.
• CISA: advisories and the catalog of known exploited vulnerabilities.
• MITRE ATT&CK: a shared language for attacker methods.