Skip to main content
Calico Enterprise 3.19 (latest) documentation

Global threat feed

A global threat feed resource (GlobalThreatFeed) represents a feed of threat intelligence used for security purposes.

Calico Enterprise supports threat feeds that give either

  • a set of IP addresses or IP prefixes, with content type IPSet, or
  • a set of domain names, with content type DomainNameSet

For each IPSet threat feed, Calico Enterprise automatically monitors flow logs for members of the set. IPSet threat feeds can also be configured to be synchronized to a global network set, allowing you to use them as a dynamically-updating deny-list by incorporating the global network set into network policy.

For each DomainNameSet threat feed, Calico Enterprise automatically monitors DNS logs for queries (QNAME) or answers (RR NAME or RDATA) that contain members of the set.

For kubectl commands, the following case-insensitive aliases may be used to specify the resource type on the CLI: globalthreatfeed.projectcalico.org, globalthreatfeeds.projectcalico.org and abbreviations such as globalthreatfeed.p and globalthreatfeeds.p.

Sample YAML​

apiVersion: projectcalico.org/v3
kind: GlobalThreatFeed
metadata:
name: sample-global-threat-feed
spec:
content: IPSet
mode: Enabled
description: "This is the sample global threat feed"
feedType: Custom
globalNetworkSet:
# labels to set on the GNS
labels:
level: high
pull:
# accepts time in golang duration format
period: 24h
http:
format:
newlineDelimited: {}
url: https://an.example.threat.feed/deny-list
headers:
- name: "Accept"
value: "text/plain"
- name: "APIKey"
valueFrom:
# secrets selected must be in the "tigera-intrusion-detection" namespace to be used
secretKeyRef:
name: "globalthreatfeed-sample-global-threat-feed-example"
key: "apikey"

Push or Pull​

You can configure Calico Enterprise to pull updates from your threat feed using a pull stanza in the global threat feed spec.

Alternately, you can have your threat feed push updates directly. Leave out the pull stanza, and configure your threat feed to create or update the Elasticsearch document that corresponds to the global threat feed object.

For IPSet threat feeds, this Elasticsearch document will be in the index .tigera.ipset.<cluster_name> and must have the ID set to the name of the global threat feed object. The doc should have a single field called ips, containing a list of IP prefixes.

For example:

PUT .tigera.ipset.cluster01/_doc/sample-global-threat-feed
{
"ips" : ["99.99.99.99/32", "100.100.100.0/24"]
}

For DomainNameSet threat feeds, this Elasticsearch document will be in the index .tigera.domainnameset.<cluster_name> and must have the ID set to the name of the global threat feed object. The doc should have a single field called domains, containing a list of domain names.

For example:

PUT .tigera.domainnameset.cluster01/_doc/example-global-threat-feed
{
"domains" : ["malware.badstuff", "hackers.r.us"]
}

Refer to the Elasticsearch document APIs for more information on how to create and update documents in Elasticsearch.

GlobalThreatFeed Definition​

Metadata​

FieldDescriptionAccepted ValuesSchema
nameThe name of this threat feed.Lower-case alphanumeric with optional -string
labelsA set of labels to apply to this threat feed.map

Spec​

FieldDescriptionAccepted ValuesSchemaDefault
contentWhat kind of threat intelligence is providedIPSet, DomainNameSetstringIPSet
modeDetermines if the threat feed is Enabled or DisabledEnabled, DisabledstringEnabled
descriptionHuman-readable description of the templateMaximum 256 charactersstring
feedTypeDistinguishes Builtin threat feeds from Custom feedsBuiltin, CustomstringCustom
globalNetworkSetInclude to sync with a global network setGlobalNetworkSetSync
pullConfigure periodic pull of threat feed updatesPull

Status​

The status is read-only for users and updated by the intrusion-detection-controller component as it processes global threat feeds.

FieldDescription
lastSuccessfulSyncTimestamp of the last successful update to the threat intelligence from the feed
lastSuccessfulSearchTimestamp of the last successful search of logs for threats
errorConditionsList of errors preventing operation of the updates or search

GlobalNetworkSetSync​

When you include a globalNetworkSet stanza in a global threat feed, it triggers synchronization with a global network set. This global network set will have the name threatfeed.<threat feed name> where <threat feed name> is the name of the global threat feed it is synced with. This is only supported for threat feeds of type IPSet.

note

A globalNetworkSet stanza only works for IPSet threat feeds, and you must also include a pull stanza.

FieldDescriptionAccepted ValuesSchema
labelsA set of labels to apply to the synced global network setmap

Pull​

When you include a pull stanza in a global threat feed, it triggers a periodic pull of new data. On successful pull and update to the data store, we update the status.lastSuccessfulSync timestamp.

If you do not include a pull stanza, you must configure your system to push updates.

FieldDescriptionAccepted ValuesSchemaDefault
periodHow often to pull an update≥ 5mDuration string24h
httpPull the update from an HTTP endpointHTTPPull

HTTPPull​

Pull updates from the threat feed by doing an HTTP GET against the given URL.

FieldDescriptionAccepted ValuesSchema
formatFormat of the data the threat feed returnsFormat
urlThe URL to querystring
headersList of additional HTTP Headers to include on the requestHTTPHeader

IPSet threat feeds must contain IP addresses or IP prefixes. For example:

 This is an IP Prefix
100.100.100.0/24
This is an address
99.99.99.99

DomainNameSet threat feeds must contain domain names. For example:

 Suspicious domains
malware.badstuff
hackers.r.us

Internationalized domain names (IDNA) may be encoded either as Unicode in UTF-8 format, or as ASCII-Compatible Encoding (ACE) according to RFC 5890.

Format​

Several different feed formats are supported. The default, newlineDelimited, expects a text file containing entries separated by newline characters. It may also include comments prefixed by #. json uses a jsonpath to extract the desired information from a JSON document. csv extracts one column from CSV-formatted data.

FieldDescriptionSchema
newlineDelimitedNewline-delimited text fileEmpty object
jsonJSON objectJSON
csvCSV fileCSV

JSON​

FieldDescriptionSchema
pathjsonpath to extract values.string

Values can be extracted from the document using any jsonpath expression, subject to the limitations mentioned below, that evaluates to a list of strings. For example: $. is valid for ["a", "b", "c"], and $.a is valid for {"a": ["b", "c"]}.

caution

No support for subexpressions and filters. Strings in brackets must use double quotes. It cannot operate on JSON decoded struct fields.

CSV​

FieldDescriptionSchema
fieldNumNumber of column containing values. Mutually exclusive with fieldName.int
fieldNameName of column containing values, requires header: true.string
headerWhether or not the document contains a header row.bool
columnDelimiterAn alternative delimiter character, such as |.string
commentDelimiterLines beginning with this character are skipped. # is common.string
recordSizeThe number of columns expected in the document. Auto detected if omitted.int
disableRecordSizeValidationDisable row size checking. Mutually exclusive with recordSize.bool

HTTPHeader​

FieldDescriptionSchema
nameHeader namestring
valueLiteral valuestring
valueFromInclude to retrieve the value from a config map or secretHTTPHeaderSource
note

You must include either value or valueFrom, but not both.

HTTPHeaderSource​

FieldDescriptionSchema
configMapKeyRefGet the value from a config mapKeyRef
secretKeyRefGet the value from a secretKeyRef

KeyRef​

KeyRef tells Calico Enterprise where to get the value for a header. The referenced Kubernetes object (either a config map or a secret) must be in the tigera-intrusion-detection namespace. The referenced Kubernetes object should have a name with following prefix format: globalthreatfeed-<GlobalThreatFeed.Name>-.

FieldDescriptionAccepted ValuesSchemaDefault
nameThe name of the config map or secretstring
keyThe key within the config map or secretstring
optionalWhether the pull can proceed without the referenced valueIf the referenced value does not exist, true means omit the header. false means abort the entire pull until it existsboolfalse