Global threat feed
A global threat feed resource (GlobalThreatFeed) represents a feed of threat intelligence used for security purposes.
Calico Enterprise supports threat feeds that give either
- a set of IP addresses or IP prefixes, with content type IPSet, or
- a set of domain names, with content type DomainNameSet
For each IPSet threat feed, Calico Enterprise automatically monitors flow logs for members of the set. IPSet threat feeds can also be configured to be synchronized to a global network set, allowing you to use them as a dynamically-updating deny-list by incorporating the global network set into network policy.
For each DomainNameSet threat feed, Calico Enterprise automatically monitors DNS logs for queries (QNAME) or answers (RR NAME or RDATA) that contain members of the set.
For kubectl
commands, the following case-insensitive aliases
may be used to specify the resource type on the CLI:
globalthreatfeed.projectcalico.org
, globalthreatfeeds.projectcalico.org
and abbreviations such as
globalthreatfeed.p
and globalthreatfeeds.p
.
Sample YAML​
apiVersion: projectcalico.org/v3
kind: GlobalThreatFeed
metadata:
name: sample-global-threat-feed
spec:
content: IPSet
mode: Enabled
description: "This is the sample global threat feed"
feedType: Custom
globalNetworkSet:
# labels to set on the GNS
labels:
level: high
pull:
# accepts time in golang duration format
period: 24h
http:
format:
newlineDelimited: {}
url: https://an.example.threat.feed/deny-list
headers:
- name: "Accept"
value: "text/plain"
- name: "APIKey"
valueFrom:
# secrets selected must be in the "tigera-intrusion-detection" namespace to be used
secretKeyRef:
name: "globalthreatfeed-sample-global-threat-feed-example"
key: "apikey"
Push or Pull​
You can configure Calico Enterprise to pull updates from your threat feed using a pull
stanza in
the global threat feed spec.
Alternately, you can have your threat feed push updates directly. Leave out the pull
stanza, and configure
your threat feed to create or update the Elasticsearch document that corresponds to the global threat
feed object.
For IPSet threat feeds, this Elasticsearch document will be in the index .tigera.ipset.<cluster_name>
and must have the ID set
to the name of the global threat feed object. The doc should have a single field called ips
, containing
a list of IP prefixes.
For example:
PUT .tigera.ipset.cluster01/_doc/sample-global-threat-feed
{
"ips" : ["99.99.99.99/32", "100.100.100.0/24"]
}
For DomainNameSet threat feeds, this Elasticsearch document will be in the index .tigera.domainnameset.<cluster_name>
and must
have the ID set to the name of the global threat feed object. The doc should have a single field called domains
, containing
a list of domain names.
For example:
PUT .tigera.domainnameset.cluster01/_doc/example-global-threat-feed
{
"domains" : ["malware.badstuff", "hackers.r.us"]
}
Refer to the Elasticsearch document APIs for more information on how to create and update documents in Elasticsearch.
GlobalThreatFeed Definition​
Metadata​
Field | Description | Accepted Values | Schema |
---|---|---|---|
name | The name of this threat feed. | Lower-case alphanumeric with optional - | string |
labels | A set of labels to apply to this threat feed. | map |
Spec​
Field | Description | Accepted Values | Schema | Default |
---|---|---|---|---|
content | What kind of threat intelligence is provided | IPSet, DomainNameSet | string | IPSet |
mode | Determines if the threat feed is Enabled or Disabled | Enabled, Disabled | string | Enabled |
description | Human-readable description of the template | Maximum 256 characters | string | |
feedType | Distinguishes Builtin threat feeds from Custom feeds | Builtin, Custom | string | Custom |
globalNetworkSet | Include to sync with a global network set | GlobalNetworkSetSync | ||
pull | Configure periodic pull of threat feed updates | Pull |
Status​
The status
is read-only for users and updated by the intrusion-detection-controller
component as
it processes global threat feeds.
Field | Description |
---|---|
lastSuccessfulSync | Timestamp of the last successful update to the threat intelligence from the feed |
lastSuccessfulSearch | Timestamp of the last successful search of logs for threats |
errorConditions | List of errors preventing operation of the updates or search |
GlobalNetworkSetSync​
When you include a globalNetworkSet
stanza in a global threat feed, it triggers synchronization
with a global network set. This global network set will have the name threatfeed.<threat feed name>
where <threat feed name>
is the name of the global threat feed it is synced with. This is only supported for
threat feeds of type IPSet.
A globalNetworkSet
stanza only works for IPSet
threat feeds, and you must also include a pull
stanza.
Field | Description | Accepted Values | Schema |
---|---|---|---|
labels | A set of labels to apply to the synced global network set | map |
Pull​
When you include a pull
stanza in a global threat feed, it triggers a periodic pull of new data. On successful
pull and update to the data store, we update the status.lastSuccessfulSync
timestamp.
If you do not include a pull
stanza, you must configure your system to push updates.
Field | Description | Accepted Values | Schema | Default |
---|---|---|---|---|
period | How often to pull an update | ≥ 5m | Duration string | 24h |
http | Pull the update from an HTTP endpoint | HTTPPull |
HTTPPull​
Pull updates from the threat feed by doing an HTTP GET against the given URL.
Field | Description | Accepted Values | Schema |
---|---|---|---|
format | Format of the data the threat feed returns | Format | |
url | The URL to query | string | |
headers | List of additional HTTP Headers to include on the request | HTTPHeader |
IPSet threat feeds must contain IP addresses or IP prefixes. For example:
This is an IP Prefix
100.100.100.0/24
This is an address
99.99.99.99
DomainNameSet threat feeds must contain domain names. For example:
Suspicious domains
malware.badstuff
hackers.r.us
Internationalized domain names (IDNA) may be encoded either as Unicode in UTF-8 format, or as ASCII-Compatible Encoding (ACE) according to RFC 5890.
Format​
Several different feed formats are supported. The default,
newlineDelimited
, expects a text file containing entries separated by
newline characters. It may also include comments prefixed by #
.
json
uses a jsonpath to extract the desired information from a
JSON document. csv
extracts one column from CSV-formatted data.
Field | Description | Schema |
---|---|---|
newlineDelimited | Newline-delimited text file | Empty object |
json | JSON object | JSON |
csv | CSV file | CSV |
JSON​
Field | Description | Schema |
---|---|---|
path | jsonpath to extract values. | string |
Values can be extracted from the document using any jsonpath
expression, subject to the limitations mentioned below, that evaluates
to a list of strings. For example: $.
is valid for ["a", "b", "c"]
,
and $.a
is valid for {"a": ["b", "c"]}
.
No support for subexpressions and filters. Strings in brackets must use double quotes. It cannot operate on JSON decoded struct fields.
CSV​
Field | Description | Schema |
---|---|---|
fieldNum | Number of column containing values. Mutually exclusive with fieldName . | int |
fieldName | Name of column containing values, requires header: true . | string |
header | Whether or not the document contains a header row. | bool |
columnDelimiter | An alternative delimiter character, such as | . | string |
commentDelimiter | Lines beginning with this character are skipped. # is common. | string |
recordSize | The number of columns expected in the document. Auto detected if omitted. | int |
disableRecordSizeValidation | Disable row size checking. Mutually exclusive with recordSize . | bool |
HTTPHeader​
Field | Description | Schema |
---|---|---|
name | Header name | string |
value | Literal value | string |
valueFrom | Include to retrieve the value from a config map or secret | HTTPHeaderSource |
You must include either value
or valueFrom
, but not both.
HTTPHeaderSource​
Field | Description | Schema |
---|---|---|
configMapKeyRef | Get the value from a config map | KeyRef |
secretKeyRef | Get the value from a secret | KeyRef |
KeyRef​
KeyRef tells Calico Enterprise where to get the value for a header. The referenced Kubernetes object
(either a config map or a secret) must be in the tigera-intrusion-detection
namespace. The referenced
Kubernetes object should have a name with following prefix format: globalthreatfeed-<GlobalThreatFeed.Name>-
.
Field | Description | Accepted Values | Schema | Default |
---|---|---|---|---|
name | The name of the config map or secret | string | ||
key | The key within the config map or secret | string | ||
optional | Whether the pull can proceed without the referenced value | If the referenced value does not exist, true means omit the header. false means abort the entire pull until it exists | bool | false |