Essentials Threat API
The information and data accessible via this API contain Proofpoint proprietary, confidential, and/or trade secret information. Sharing or providing the information to another party without Proofpoint's express written consent is prohibited. Please review the updated Terms of Use for Proofpoint APIs. The API's Terms of Use can be found online at API Terms of Use | Proofpoint US.
Overview
Security Information and Event Management (SIEM) solutions are used by many organizations to identify and correlate various security events occurring in their point products. Examples of SIEM products include HP's ArcSight, IBM's QRadar, and Splunk.
The API allows integration with these solutions by giving administrators the ability to periodically download detailed information about several types of threat events in a SIEM-compatible, vendor-neutral format. Currently, the following event types are exposed:
- Blocked or permitted clicks to threats recognized by URL Defense
- Blocked or delivered messages that contain threats recognized by URL Defense or Attachment Defense
API Features
General Service Notes
- All timestamps in the returned events are in UTC.
- The API is returns data in JSON format.
- A maximum of one hour of data can be requested in a single transaction.
- No paging support is available; all the applicable events in the requested time period will be returned in the log.
- The results provided by this API may not be in any logical order.
- Requests to the service may be throttled to prevent abuse.
- If results cannot be obtained within a timeout period, the service will return an error.
- The maximum time into the past that can be queried is 3 days with a maximum fetch time of 1 hour.
Security
Each request must use:
- SSL.
- Integration Keys to authenticate to the API.
- the HTTP Basic Authorization method.
- the HTTP GET method.
Standard Responses
Requests to the endpoints can produce a response with a variety of HTTP status codes. The following table describes the scenarios in which these codes can be produced.
Code | Message | Scenarios |
---|---|---|
200 |
Success |
The request was successful. If records matching are found, they will be returned in the response body. |
400 |
Bad Request |
The request is missing a mandatory "request" parameter, a parameter contains data which is incorrectly formatted, or the API doesn't have enough information to determine the identity of the customer. |
401 |
Unauthorized |
There is no authorization information included in the request, the authorization information is incorrect, or the user is not authorized. |
403 |
Forbidden |
The user is authenticated for the service but is not authorized to access data for the given customer. |
429 |
Too Many Requests |
The user has made too many requests over the past 24 hours and has been throttled. |
500 |
Internal Server Error |
The service has encountered an unexpected situation and is unable to give a better response to the request. |
Throttle Limits
These endpoints provide methods to fetch information about click and message events for a given time period. The number of queries connected to this resource are limited by a simple, rolling 24-hour throttle. Once exceeded, the API will start returning 429 HTTP status codes until 24 hours past the oldest request has elapsed. Requests to the clicks/permitted API and requests to other APIs are throttled into different pools. This allows more frequent queries to the clicks/permitted API.
Endpoint |
Max Number of Requests |
---|---|
clicks/permitted |
1800 per 24 hours |
all issues clicks/blocked messages/delivered messages/blocked |
1800 per 24 hours |
Available Endpoints
All endpoints are available on one of two hosts, depending on region:
/v2/siem/clicks/blocked
Fetch events for clicks to malicious URLs blocked in the specified time period
/v2/siem/clicks/permitted
Fetch events for clicks to malicious URLs permitted in the specified time period
/v2/siem/messages/blocked
Fetch events for messages blocked in the specified time period which contained a known threat
/v2/siem/messages/delivered
Fetch events for messages delivered in the specified time period which contained a known threat
/v2/siem/issues
Fetch events for clicks to malicious URLs permitted and messages delivered containing a known threat within the specified time period. (It is a combination of /v2/siem/clicks/permitted and /v2/siem/messages/delivered)
/v2/siem/all
Fetch events for all clicks and messages relating to known threats within the specified time period
Required Parameters
One of the following three query parameters describing the desired time range for the data must be supplied with each request:
interval
A string containing an ISO8601-formatted interval. If this interval overlaps with previous requests for data, records from the previous request may be duplicated. The minimum interval is thirty seconds. The maximum interval is one hour.
Examples:
- 2016-05-01T12:00:00Z/2016-05-01T13:00:00Z - an hour interval, beginning at noon UTC on 05-01-2016
- PT30M/2016-05-01T12:30:00Z - the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC
- 2016-05-01T05:00:00-0700/PT30M - the same interval as above, but using -0700 as the time zone
sinceSeconds
An integer representing a time window in seconds from the current API server time. The start of the window is the current API server time, rounded to the nearest minute, less the number of seconds provided. The end of the window is the current API server time rounded to the nearest minute. If JSON output is selected, the end time is included in the returned result.
sinceTime
A string containing an ISO8601 date. It represents the start of the data retrieval period. The end of the period is determined by current API server time rounded to the nearest minute. If JSON output is selected, the end time is included in the returned result.
Optional Parameters
One or more of these parameters may also be provided:
customerData
A boolean specifying whether customer threat data is returned. All customers underneath the partner will be included. If no value is specified, defaults to False. Either customerData or ownData MUST be set to True.
ownData
A boolean specifying whether the organization's own threat data is returned. If no value is specified, defaults to True. Either customerData or ownData MUST be set to True.
threatType
A string specifying which threat type will be returned in the data. If no value is specified, all threat types are returned. The following values are accepted:
- url
- attachment
- messageText
threatStatus
A string specifying which threat statuses will be returned in the data. If no value is specified, active and cleared threats are returned. The following values are accepted:
- active
- cleared
- falsePositive
Example Commands in Curl
The following commands assume that principal and secret are defined environment variables. They correspond to the service principal and secret that was created on the API Keys page.
curl "https://us-siem.proofpointessentials.com/v2/siem/issues?format=json&sinceTime=2016-05-01T12:00:00Z" --user "$PRINCIPAL:$SECRET" -s
Retrieves events from noon on 05/01/2016 to the present. Returned events are limited to just permitted clicks and delivered messages with known threats.
curl "https://us-siem.proofpointessentials.com/v2/siem/all?sinceSeconds=3600" --user "$PRINCIPAL:$SECRET" -s
Retrieves events to the present, starting 3600 seconds before the query time. All events are returned.
curl "https://us-siem.proofpointessentials.com/v2/siem/clicks/permitted?interval=PT30M/2016-05-01T12:30:00Z" --user "$PRINCIPAL:$SECRET" -s
Retrieves events from the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC. Only permitted clicks are returned.
curl "https://us-siem.proofpointessentials.com/v2/siem/all?interval=PT30M/2016-05-01T12:30:00Z&threatStatus=falsePositive&threatStatus=active&threatStatus=cleared" --user "$PRINCIPAL:$SECRET" -s
Retrieves events from the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC. All events are returned. False positives are included in the output.
Note About Event Time
Event Time in SIEM is the time the event was ingested into the SIEM. The time range used in the query parameters controls which events the SIEM API returns based on the time that the event was created, not the time the event occurred. The time an event is created is always after either of these two times:
-
the time that the message was sent or the time click occurred (messageTime or clickTime)
-
the time that the threat referenced by the message or click was recognized and condemned by Proofpoint (threatsInfoMap/threatTime)
In other words, a request using the sinceSeconds=3600 parameter will retrieve all events which have been created in the last hour. It is possible that the events returned from that interval reference messages or clicks which were first observed more than one hour ago—perhaps even several days ago.
Results
Structure
Fields |
Type |
Restrictions |
Description |
---|---|---|---|
queryEndTime |
DateTime |
ISO8601 Timestamp |
The time at which the period queried for data ended |
messagesDelivered |
Array |
Message Events |
An array containing all messages with threats which were delivered by PPS |
messagesBlocked |
Array |
Message Events |
An array containing all messages with threats which were quarantined by PPS |
clicksPermitted |
Array |
Click Events |
An array containing all clicks to URL threats which were permitted |
clicksBlocked |
Array |
Click Events |
An array containing all clicks to URL threats which were blocked |
Message Events
Field |
Content |
Restrictions |
Description |
---|---|---|---|
ccAddresses |
Array of strings |
|
A list of email addresses contained within the CC: header, excluding friendly names. |
stackName |
String |
|
The name of the Essentials stack which processed the message. |
customerEid |
String |
|
The customers entity ID. |
customerName |
String |
|
The customer's name, as configured in Essentials. |
parentEid |
String |
|
The parent's EID. |
parentName |
String |
|
The parent's name, as configured in Essentials. |
completelyRewritten |
String |
|
The rewrite status of the message. If value is 'true', all instances of URL threats within the message were successfully rewritten. If the value is 'false', at least one instance of the a threat URL was not rewritten. If the value is 'na', the message did not contain any URL-based threats. |
fromAddress |
String |
|
The email address contained in the From: header, excluding friendly name. |
GUID |
String |
|
The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique. |
headerFrom |
String |
|
The full content of the From: header, including any friendly name. |
headerReplyTo |
String |
|
If present, the full content of the Reply-To: header, including any friendly names. |
impostorScore |
int |
0-100 |
The impostor score of the message. Higher scores indicate higher certainty. |
malwareScore |
int |
0-100 |
The malware score of the message. Higher scores indicate higher certainty. |
messageDetailsUrl |
String |
|
A permalink to the messages' details page. |
messageID |
String |
|
Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique. |
messageParts |
Array of JSON Structures |
|
An array of structures which contain details about parts of the message, including both message bodies and attachments. |
messageParts/contentType |
String |
|
The true, detected Content-Type of the messagePart. This may differ from the oContentType value. |
messageParts/disposition |
String |
|
If the value is "inline," the messagePart is a message body. If the value is "attached," the messagePart is an attachment. |
messageParts/filename |
String |
|
The filename of the messagePart. |
messageParts/md5 |
String |
|
The MD5 hash of the messagePart contents. |
messageParts/oContentType |
String |
|
The declared Content-Type of the messagePart. |
messageParts/sandboxStatus |
String |
|
The verdict returned by the sandbox during the scanning process. If the value is "unsupported", the messagePart is not supported by Attachment Defense and was not scanned. If the value is "clean", the sandbox returned a clean verdict. If the value is "threat", the sandbox returned a malicious verdict. If the value is "prefilter", the messagePart contained no active content, and was therefore not sent to the sandboxing service. If the value is "uploaded," the message was uploaded by PPS to the sandboxing service, but did not yet have a verdict at the time the message was processed. If the value is "inprogress," the attachment had been uploaded and was awaiting scanning at the time the message was processed. If the verdict is "uploaddisabled," the attachment was eligible for scanning, but was not uploaded because of PPS policy. |
messageParts/sha256 |
String |
|
The SHA256 hash of the messagePart contents. |
messageSize |
Integer |
|
The size in bytes of the message, including headers and attachments. |
messageTime |
DateTime |
ISO8601 Timestamp |
When the message was delivered to the user or quarantined by PPS |
phishScore |
int |
0-100 |
The phish score of the message. Higher scores indicate higher certainty. |
quarantineRule |
String |
|
The name of the rule which quarantined the message. This appears only for messagesBlocked events. |
recipient |
String |
Email Address |
An array containing the email addresses of the SMTP (envelope) recipients |
replyToAddress |
String |
|
The email address contained in the Reply-To: header, excluding friendly name. |
sender |
String |
Email Address |
The email address of the SMTP (envelope) sender. The user-part is hashed. The domain-part is cleartext. |
senderIP |
String |
IP Address |
The IP address of the sender. |
spamScore |
int |
0-100 |
The spam score of the message. Higher scores indicate higher certainty. |
subject |
String |
|
The subject line of the message, if available. |
threatsInfoMap |
Array of JSON Structures |
|
An array of structures which contain details about detected threats within the message. There may be more than one threat per message. |
threatsInfoMap/detectionType |
String |
Nullable |
New field detectionType and ThreatInfo added to response |
threatsInfoMap/classification |
String |
|
The category of threat found in the message |
threatsInfoMap/threat |
String |
|
The artifact which was condemned by Proofpoint. The malicious URL, hash of the attachment threat, or email address of the impostor sender. |
threatsInfoMap/threatId |
String |
|
The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints. |
threatsInfoMap/threatStatus |
String |
|
The current state of the threat. |
threatsInfoMap/threatTime |
DateTime |
ISO8601 Timestamp |
Proofpoint assigned the threatStatus at this time. |
threatsInfoMap/threatType |
String |
|
Whether the threat was an attachment, URL, or message type. |
threatsInfoMap/actors |
Array of JSON Structures |
|
An array of structures which contain details about the actors associated with a threat |
toAddresses |
Array of strings |
|
A list of email addresses contained within the To: header, excluding friendly names. |
xmailer |
String |
|
The content of the X-Mailer: header, if present. |
Click Events
Field |
Content |
Restrictions |
Description |
---|---|---|---|
classification |
String |
|
The threat category of the malicious URL |
clickIP |
String |
IP Address |
The external IP address of the user who clicked on the link. If the user is behind a firewall performing network address translation, the IP address of the firewall will be shown. |
clickTime |
DateTime |
ISO8601 Timestamp |
The time the user clicked on the URL |
GUID |
String |
|
The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique. |
id |
String |
|
The unique id of the click. |
recipient |
String |
Email Address |
The email address of the recipient |
sender |
String |
Email Address |
The email address of the sender. The user-part is hashed. The domain-part is cleartext. |
senderIP |
String |
IP Address |
The IP address of the sender |
threatID |
String |
|
The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints. |
threatTime |
DateTime |
ISO8601 Timestamp |
Proofpoint identified the URL as a threat at this time. |
threatStatus |
String |
|
The current state of the threat. |
url |
String |
URL |
The malicious URL which was clicked |
userAgent |
String |
|
The User-Agent header from the clicker's HTTP request |
Example Output
{ "clicksPermitted": [{ "classification": "MALWARE", "clickIP": "192.0.2.1", "clickTime": "2016-06-24T19:17:44.000Z", "GUID": "b27dbea0-87d5-463b-b93c-4e8b708289ce", "id": "8c8b4895-a277-449f-r797-547e3c89b25a", "messageID": "8c6cfedd-3050-4d65-8c09-c5f65c38da81", "recipient": "bruce.wayne@pharmtech.zz", "sender": "9facbf452def2d7efc5b5c48cdb837fa@badguy.zz", "senderIP": "192.0.2.255", "threatID": "61f7622167144dba5e3ae4480eeee78b23d66f7dfed970cfc3d086cc0dabdf50", "threatTime": "2016-06-24T19:17:46.000Z", "threatStatus": "active", "url": "http://badguy.zz/", "userAgent": "Mozilla/5.0(WindowsNT6.1;WOW64;rv:27.0)Gecko/20100101Firefox/27.0" } ], "messagesBlocked": [{ "GUID": "c26dbea0-80d5-463b-b93c-4e8b708219ce", "QID": "r2FNwRHF004109", "ccAddresses": ["bruce.wayne@university-of-education.zz"], "completelyRewritten": "true", "fromAddress": "badguy@evil.zz", "headerCC": "\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>", "headerFrom": "\"A. Badguy\" <badguy@evil.zz>", "headerReplyTo": null, "headerTo": "\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>", "impostorScore": 0, "malwareScore": 100, "messageID": "20160624211145.62086.mail@evil.zz", "messageParts": [{ "contentType": "text/plain", "disposition": "inline", "filename": "text.txt", "md5": "008c5926ca861023c1d2a36653fd88e2", "oContentType": "text/plain", "sandboxStatus": "unsupported", "sha256": "85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281" }, { "contentType": "application/pdf", "disposition": "attached", "filename": "Invoice for Pharmtech.pdf", "md5": "5873c7d37608e0d49bcaa6f32b6c731f", "oContentType": "application/pdf", "sandboxStatus": "threat", "sha256": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca" } ], "messageTime": "2016-06-24T21:18:38.000Z", "modulesRun": ["pdr", "sandbox", "spam", "urldefense"], "phishScore": 46, "policyRoutes": ["default_inbound", "executives"], "quarantineFolder": "Attachment Defense", "quarantineRule": "module.sandbox.threat", "recipient": ["clark.kent@pharmtech.zz", "diana.prince@pharmtech.zz"], "replyToAddress": null, "sender": "e99d7ed5580193f36a51f597bc2c0210@evil.zz", "senderIP": "192.0.2.255", "spamScore": 4, "subject": "Please find a totally safe invoice attached.", "threatsInfoMap": [{ "classification": "MALWARE", "threat": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca", "threatId": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca", "threatStatus": "active", "threatTime": "2016-06-24T21:18:38.000Z", "threatType": "ATTACHMENT", }, { "classification": "MALWARE", "threat": "badsite.zz", "threatId": "3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa", "threatTime": "2016-06-24T21:18:07.000Z", "threatType": "URL", } ], "toAddresses": ["clark.kent@pharmtech.zz", "diana.prince@pharmtech.zz"] }, "xmailer": "Spambot v2.5"], "queryEndTime": "2016-06-24T21:36:00Z" }