SIEM API - Proofpoint, Inc.

General Service Notes

All timestamps in the returned events are in UTC.
The API is designed to support different SIEM-compatible formats: Syslog and JSON.
A maximum of one hour of data can be requested in a single transaction.
No paging support is available; all the applicable events in the requested time period will be returned in the log.
The results provided by this API may not be in any logical order.
Requests to the service may be throttled to prevent abuse.
If results cannot be obtained within a timeout period, the service will return an error.
The maximum time into the past that can be queried is 7 days with a maximum fetch time of 1 hour.

Security

Each request must use:

SSL.
service credentials to authenticate to the API.
the HTTP Basic Authorization method.
the HTTP GET method.

Standard Responses

Requests to the endpoints can produce a response with a variety of HTTP status codes. The following table describes the scenarios in which these codes can be produced.

Code	Message	Scenarios
200	Success	At least one record matching the specified criteria was found and returned in the response body. In the case of a JSON format, the structure is always returned, even if empty.
204	No Content	Syslog format only: If no records matching the specified criteria were found, a status code of 204 will be returned with empty content.
400	Bad Request	The request is missing a mandatory "request" parameter, a parameter contains data which is incorrectly formatted, or the API doesn't have enough information to determine the identity of the customer.
401	Unauthorized	There is no authorization information included in the request, the authorization information is incorrect, or the user is not authorized.
403	Forbidden	The user is authenticated for the service but is not authorized to access data for the given customer.
429	Too Many Requests	The user has made too many requests over the past 24 hours and has been throttled.
500	Internal Server Error	The service has encountered an unexpected situation and is unable to give a better response to the request.

Throttle Limits

These endpoints provide methods to fetch information about click and message events for a given time period. The number of queries connected to this resource are limited by a simple, rolling 24-hour throttle. Once exceeded, the API will start returning 429 HTTP status codes until 24 hours past the oldest request has elapsed. Requests to the clicks/permitted API and requests to other APIs are throttled into different pools. This allows more frequent queries to the clicks/permitted API.

Endpoint	Max Number of Requests
clicks/permitted	1800 per 24 hours
all issues clicks/blocked messages/delivered messages/blocked	1800 per 24 hours

Endpoint

Max Number of Requests

clicks/permitted

1800 per 24 hours

all

issues

clicks/blocked

messages/delivered

messages/blocked

1800 per 24 hours

Available Endpoints

All endpoints are available on the tap-api-v2.proofpoint.com host—for example, https://tap-api-v2.proofpoint.com/v2...clicks/blocked.

/v2/siem/clicks/blocked

Fetch events for clicks to malicious URLs blocked in the specified time period

/v2/siem/clicks/permitted

Fetch events for clicks to malicious URLs permitted in the specified time period

/v2/siem/messages/blocked

Fetch events for messages blocked in the specified time period which contained a known threat

/v2/siem/messages/delivered

Fetch events for messages delivered in the specified time period which contained a known threat

/v2/siem/issues

Fetch events for clicks to malicious URLs permitted and messages delivered containing a known threat within the specified time period. (It is a combination of /v2/siem/clicks/permitted and /v2/siem/messages/delivered)

/v2/siem/all

Fetch events for all clicks and messages relating to known threats within the specified time period

Required Parameters

One of the following three query parameters describing the desired time range for the data must be supplied with each request:

interval

A string containing an ISO8601-formatted interval. If this interval overlaps with previous requests for data, records from the previous request may be duplicated. The minimum interval is thirty seconds. The maximum interval is one hour.

Examples:

2016-05-01T12:00:00Z/2016-05-01T13:00:00Z - an hour interval, beginning at noon UTC on 05-01-2016
PT30M/2016-05-01T12:30:00Z - the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC
2016-05-01T05:00:00-0700/PT30M - the same interval as above, but using -0700 as the time zone

sinceSeconds

An integer representing a time window in seconds from the current API server time. The start of the window is the current API server time, rounded to the nearest minute, less the number of seconds provided. The end of the window is the current API server time rounded to the nearest minute. If JSON output is selected, the end time is included in the returned result.

sinceTime

A string containing an ISO8601 date. It represents the start of the data retrieval period. The end of the period is determined by current API server time rounded to the nearest minute. If JSON output is selected, the end time is included in the returned result.

Optional Parameters

One or more of these parameters may also be provided:

format

A string specifying the format in which data is returned. If no format is specified, syslog will be used as the default. The following values are accepted:

syslog
JSON

threatType

A string specifying which threat type will be returned in the data. If no value is specified, all threat types are returned. The following values are accepted:

url
attachment
messageText

threatStatus

A string specifying which threat statuses will be returned in the data. If no value is specified, active and cleared threats are returned. The following values are accepted:

active
cleared
falsePositive

Example Commands in Curl

The following commands assume that principal and secret are defined environment variables. They correspond to the service principal and secret that was created on the Settings page.

curl "https://tap-api-v2.proofpoint.com/v2/siem/issues?format=json&sinceTime=2016-05-01T12:00:00Z" --user "$PRINCIPAL:$SECRET" -s

Retrieves events from noon on 05/01/2016 to the present. Returned events are limited to just permitted clicks and delivered messages with known threats. Output is in the JSON format.

curl "https://tap-api-v2.proofpoint.com/v2/siem/all?format=syslog&sinceSeconds=3600" --user "$PRINCIPAL:$SECRET" -s

Retrieves events to the present, starting 3600 seconds before the query time. All events are returned. Output is in the syslog Format.

curl "https://tap-api-v2.proofpoint.com/v2/siem/clicks/permitted?format=syslog&interval=PT30M/2016-05-01T12:30:00Z" --user "$PRINCIPAL:$SECRET" -s

Retrieves events from the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC. Only permitted clicks are returned. Output is in the syslog format.

curl "https://tap-api-v2.proofpoint.com/v2/siem/all?format=json&interval=PT30M/2016-05-01T12:30:00Z&threatStatus=falsePositive&threatStatus=active&threatStatus=cleared" --user "$PRINCIPAL:$SECRET" -s

Retrieves events from the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC. All events are returned. False positives are included in the output. Output is in the JSON format.

Note About Event Time

Event Time in SIEM is the time the event was ingested into the SIEM. The time range used in the query parameters controls which events the SIEM API returns based on the time that the event was created, not the time the event occurred. The time an event is created is always after either of these two times:

the time that the message was sent or the time click occurred (messageTime or clickTime)
the time that the threat referenced by the message or click was recognized and condemned by Proofpoint (threatsInfoMap/threatTime)

In other words, a request using the sinceSeconds=3600 parameter will retrieve all events which have been created in the last hour. It is possible that the events returned from that interval reference messages or clicks which were first observed more than one hour ago—perhaps even several days ago.

Results

JSON

If the JSON output is used, the following structure will always be produced, even if there are no events inside any individual (or all) event arrays.

Structure

Fields	Type	Restrictions	Description
queryEndTime	DateTime	ISO8601 Timestamp	The time at which the period queried for data ended
messagesDelivered	Array	Message Events	An array containing all messages with threats which were delivered by PPS
messagesBlocked	Array	Message Events	An array containing all messages with threats which were quarantined by PPS
clicksPermitted	Array	Click Events	An array containing all clicks to URL threats which were permitted
clicksBlocked	Array	Click Events	An array containing all clicks to URL threats which were blocked

Message Events

Field	Content	Restrictions	Description
ccAddresses	Array of strings		A list of email addresses contained within the CC: header, excluding friendly names.
clusterId	String		The name of the PPS cluster which processed the message.
completelyRewritten	String	true false na	The rewrite status of the message. If value is 'true', all instances of URL threats within the message were successfully rewritten. If the value is 'false', at least one instance of the a threat URL was not rewritten. If the value is 'na', the message did not contain any URL-based threats.
fromAddress	String		The email address contained in the From: header, excluding friendly name.
GUID	String		The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
headerFrom	String		The full content of the From: header, including any friendly name.
headerReplyTo	String		If present, the full content of the Reply-To: header, including any friendly names.
impostorScore	int	0-100	The impostor score of the message. Higher scores indicate higher certainty.
malwareScore	int	0-100	The malware score of the message. Higher scores indicate higher certainty.
messageID	String		Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique.
messageParts	Array of JSON Structures		An array of structures which contain details about parts of the message, including both message bodies and attachments.
messageParts/contentType	String		The true, detected Content-Type of the messagePart. This may differ from the oContentType value.
messageParts/disposition	String	inline attached	If the value is "inline," the messagePart is a message body. If the value is "attached," the messagePart is an attachment.
messageParts/filename	String		The filename of the messagePart.
messageParts/md5	String		The MD5 hash of the messagePart contents.
messageParts/oContentType	String		The declared Content-Type of the messagePart.
messageParts/sandboxStatus	String	unsupported threat clean prefilter uploaded inprogress uploaddisabled	The verdict returned by the sandbox during the scanning process. If the value is "unsupported", the messagePart is not supported by Attachment Defense and was not scanned. If the value is "clean", the sandbox returned a clean verdict. If the value is "threat", the sandbox returned a malicious verdict. If the value is "prefilter", the messagePart contained no active content, and was therefore not sent to the sandboxing service. If the value is "uploaded," the message was uploaded by PPS to the sandboxing service, but did not yet have a verdict at the time the message was processed. If the value is "inprogress," the attachment had been uploaded and was awaiting scanning at the time the message was processed. If the verdict is "uploaddisabled," the attachment was eligible for scanning, but was not uploaded because of PPS policy.
messageParts/sha256	String		The SHA256 hash of the messagePart contents.
messageSize	Integer		The size in bytes of the message, including headers and attachments.
messageTime	DateTime	ISO8601 Timestamp	When the message was delivered to the user or quarantined by PPS
modulesRun	Array of strings		The list of PPS modules which processed the message.
phishScore	int	0-100	The phish score of the message. Higher scores indicate higher certainty.
policyRoutes	Array of strings		The policy routes that the message matched during processing by PPS.
QID	String		The queue ID of the message within PPS. It can be used to identify the message in PPS and is not unique.
quarantineFolder	String		The name of the folder which contains the quarantined message. This appears only for messagesBlocked.
quarantineRule	String		The name of the rule which quarantined the message. This appears only for messagesBlocked events.
recipient	String	Email Address	An array containing the email addresses of the SMTP (envelope) recipients
replyToAddress	String		The email address contained in the Reply-To: header, excluding friendly name.
sender	String	Email Address	The email address of the SMTP (envelope) sender. The user-part is hashed. The domain-part is cleartext.
senderIP	String	IP Address	The IP address of the sender.
spamScore	int	0-100	The spam score of the message. Higher scores indicate higher certainty.
subject	String		The subject line of the message, if available.
threatsInfoMap	Array of JSON Structures		An array of structures which contain details about detected threats within the message. There may be more than one threat per message.
threatsInfoMap/detectionType	String	Nullable	New field detectionType and ThreatInfo added to response
threatsInfoMap/campaignId	String		An identifier for the campaign of which the threat is a member, if available at the time of the query. Threats can be linked to campaigns even after these events are retrieved.
threatsInfoMap/classification	String	Malware Phish Spam Impostor (for BEC/Message Text threats) TOAD (Telephone-Oriented Attack Delivery)	The category of threat found in the message
threatsInfoMap/threat	String	SHA256 URL email address	The artifact which was condemned by Proofpoint. The malicious URL, hash of the attachment threat, or email address of the impostor sender.
threatsInfoMap/threatId	String		The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.
threatsInfoMap/threatStatus	String	active falsepositive cleared	The current state of the threat.
threatsInfoMap/threatTime	DateTime	ISO8601 Timestamp	Proofpoint assigned the threatStatus at this time.
threatsInfoMap/threatType	String	Attachment URL Message	Whether the threat was an attachment, URL, or message type.
threatsInfoMap/threatUrl	String	URL	A link to the entry about the threat on the TAP Dashboard
toAddresses	Array of strings		A list of email addresses contained within the To: header, excluding friendly names.
xmailer	String		The content of the X-Mailer: header, if present.

Click Events

Field	Content	Restrictions	Description
campaignId	String		An identifier for the campaign of which the threat is a member, if available at the time of the query. Threats can be linked to campaigns even after these events are retrieved.
classification	String	Malware Phish Spam	The threat category of the malicious URL
clickIP	String	IP Address	The external IP address of the user who clicked on the link. If the user is behind a firewall performing network address translation, the IP address of the firewall will be shown.
clickTime	DateTime	ISO8601 Timestamp	The time the user clicked on the URL
GUID	String		The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
id	String		The unique id of the click.
recipient	String	Email Address	The email address of the recipient
sender	String	Email Address	The email address of the sender. The user-part is hashed. The domain-part is cleartext.
senderIP	String	IP Address	The IP address of the sender
threatID	String		The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.
threatTime	DateTime	ISO8601 Timestamp	Proofpoint identified the URL as a threat at this time.
threatURL	String	URL	A link to the entry on the TAP Dashboard for the particular threat
threatStatus	String	active falsepositive cleared	The current state of the threat.
url	String	URL	The malicious URL which was clicked
userAgent	String		The User-Agent header from the clicker's HTTP request

Example Output

{
   "clicksPermitted":[
      {
         "campaignId":"46e01b8a-c899-404d-bcd9-189bb393d1a7",
         "classification":"MALWARE",
         "clickIP":"192.0.2.1",
         "clickTime":"2016-06-24T19:17:44.000Z",
         "GUID": "b27dbea0-87d5-463b-b93c-4e8b708289ce",
         "id": "8c8b4895-a277-449f-r797-547e3c89b25a",
         "messageID":"8c6cfedd-3050-4d65-8c09-c5f65c38da81",
         "recipient":"bruce.wayne@pharmtech.zz",
         "sender":"9facbf452def2d7efc5b5c48cdb837fa@badguy.zz",
         "senderIP":"192.0.2.255",
         "threatID":"61f7622167144dba5e3ae4480eeee78b23d66f7dfed970cfc3d086cc0dabdf50",
         "threatTime":"2016-06-24T19:17:46.000Z",
         "threatURL":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/61f7622167144dba5e3ae4480eeee78b23d66f7dfed970cfc3d086cc0dabdf50",
         "threatStatus": "active",
         "url":"http://badguy.zz/",
         "userAgent":"Mozilla/5.0(WindowsNT6.1;WOW64;rv:27.0)Gecko/20100101Firefox/27.0"
      }
   ],
   "messagesBlocked":[
      {
         "GUID":"c26dbea0-80d5-463b-b93c-4e8b708219ce",
         "QID":"r2FNwRHF004109",
         "ccAddresses":[
            "bruce.wayne@university-of-education.zz"
         ],
         "clusterId":"pharmtech_hosted",
         "completelyRewritten":"true",
         "fromAddress":"badguy@evil.zz",
         "headerCC":"\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>",
         "headerFrom":"\"A. Badguy\" <badguy@evil.zz>",
         "headerReplyTo":null,
         "headerTo":"\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>",
         "impostorScore":0,
         "malwareScore":100,
         "messageID":"20160624211145.62086.mail@evil.zz",
         "messageParts":[
            {
               "contentType":"text/plain",
               "disposition":"inline",
               "filename":"text.txt",
               "md5":"008c5926ca861023c1d2a36653fd88e2",
               "oContentType":"text/plain",
               "sandboxStatus":"unsupported",
               "sha256":"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281"
            },
            {
               "contentType":"application/pdf",
               "disposition":"attached",
               "filename":"Invoice for Pharmtech.pdf",
               "md5":"5873c7d37608e0d49bcaa6f32b6c731f",
               "oContentType":"application/pdf",
               "sandboxStatus":"threat",
               "sha256":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
            }
         ],
         "messageTime":"2016-06-24T21:18:38.000Z",
         "modulesRun":[
            "pdr",
            "sandbox",
            "spam",
            "urldefense"
         ],
         "phishScore":46,
         "policyRoutes":[
            "default_inbound",
            "executives"
         ],
         "quarantineFolder":"Attachment Defense",
         "quarantineRule":"module.sandbox.threat",
         "recipient":[
            "clark.kent@pharmtech.zz",
            "diana.prince@pharmtech.zz"
         ],
         "replyToAddress":null,
         "sender":"e99d7ed5580193f36a51f597bc2c0210@evil.zz",
         "senderIP":"192.0.2.255",
         "spamScore":4,
         "subject":"Please find a totally safe invoice attached.",
         "threatsInfoMap":[
            {
               "campaignId":"46e01b8a-c899-404d-bcd9-189bb393d1a7",
               "classification":"MALWARE",
               "threat":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
               "threatId":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
               "threatStatus":"active",
               "threatTime":"2016-06-24T21:18:38.000Z",
               "threatType":"ATTACHMENT",
               "threatUrl":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
            },
            {
               "campaignId":"46e01b8a-c899-404d-bcd9-189bb393d1a7",
               "classification":"MALWARE",
               "threat":"badsite.zz",
               "threatId":"3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa",
               "threatTime":"2016-06-24T21:18:07.000Z",
               "threatType":"URL",
               "threatUrl":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa"
            }
         ],
         "toAddresses":[
            "clark.kent@pharmtech.zz",
            "diana.prince@pharmtech.zz"
         ]
      },
      "xmailer":"Spambot v2.5"
   ],
   "queryEndTime":"2016-06-24T21:36:00Z"
}

syslog

Individual events are CRLF-delimited. Events are produced in the syslog format, as described by RFC 5424. All data is contained within the structured-data field. The freeform MSG field is blank.

Structure

Event Type (MSGID)	Priority Value	SD-ID	Restrictions	Description
MSGDLV	<36> (Warning)	tapmsg@21139	Message Events	A message containing a threat was delivered by PPS.
MSGBLK	<38> (Informational)	tapmsg@21139	Message Events	A message containing a threat was quarantined by PPS.
CLKPER	<33> (Alert)	tapclk@21139	Click Events	A click to a threat was permitted.
CLKBLK	<38> (Informational)	tapclk@21139	Click Events	A click to a threat was blocked.

Message Event Data

Field	Content	Restrictions	Description
ccAddresses	String	Comma-delimited	A list of email addresses contained within the CC: header, excluding friendly names.
clusterId	String		The name of the PPS cluster which processed the message.
completelyRewritten	String	true false na	The rewrite status of the message. If value is 'true', all instances of URL threats within the message were successfully rewritten. If the value is 'false', at least one instance of the a threat URL was not rewritten. If the value is 'na', the message did not contain any URL-based threats.
fromAddress	String		The email address contained in the From: header, excluding friendly name.
GUID	String		The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
headerFrom	String		The full content of the From: header, including any friendly name.
headerReplyTo	String		If present, the full content of the Reply-To: header, including any friendly names.
impostorScore	int	0-100	The impostor score of the message. Higher scores indicate higher certainty.
malwareScore	int	0-100	The malware score of the message. Higher scores indicate higher certainty.
messageID	String		Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique.
messageParts	Map	JSON Structure	An array of structures which contain details about parts of the message, including both message bodies and attachments. The structure is exactly the same as the above.
messageSize	Integer		The size in bytes of the message, including headers and attachments.
messageTime	DateTime	ISO8601 Timestamp	When the message was delivered to the user or quarantined by PPS
modulesRun	String	Comma-delimited	The list of PPS modules which processed the message.
phishScore	int	0-100	The phish score of the message. Higher scores indicate higher certainty.
policyRoutes	Strings	Comma-delimited	The policy routes that the message matched during processing by PPS.
QID	String		The queue ID of the message within PPS. It can be used to identify the message in PPS and is not unique.
quarantineFolder	String		The name of the folder which contains the quarantined message. This appears only for messagesBlocked.
quarantineRule	String		The name of the rule which quarantined the message. This appears only for messagesBlocked events.
recipient	String	Email Address	An array containing the email addresses of the recipients
replyToAddress	String		The email address contained in the Reply-To: header, excluding friendly name.
sender	String	Email Address	The email address of the sender. The user-part is hashed. The domain-part is cleartext.
senderIP	String	IP Address	The IP Address of the sender
spamScore	int	0-100	The spam score of the message. Higher scores indicate higher certainty.
subject	String		The subject line of the message, if available.
threatsInfoMap	Map	JSON Structure	A string containing a JSON structure with details about detected threats within the message. There may be more than one threat per message. The threatsInfoMap structure is exactly the same as the JSON output above.
toAddresses	String	Comma-delimited	A list of email addresses contained within the To: header, excluding friendly names.
xmailer	String		The content of the X-Mailer: header, if present.

Click Event Data

Field	Content	Restrictions	Description
campaignId	String		An identifier for the campaign of which the threat is a member, if available at the time of the query. Threats can be linked to campaigns even after these events are retrieved.
class	String	Malware Phish Spam	The threat category of the malicious URL
clickIP	String	IP Address	The external IP address of the user who clicked on the link. If the user is behind a firewall performing network address translation, the IP address of the firewall will be shown.
clickTime	DateTime	ISO8601 Timestamp	The time the user clicked on the URL
messageID	String		Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique.
GUID	String		The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
recipient	String	Email Address	The email address of the recipient
sender	String	Email Address	The email address of the sender. The user-part is hashed. The domain-part is cleartext.
senderIP	String	IP Address	The IP address of the sender
threatID	String		The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.
threatTime	DateTime	ISO8601 Timestamp	Proofpoint identified the URL as a threat at this time.
threatURL	String	URL	A link to the entry on the TAP Dashboard for the particular threat.
url	String	URL	The malicious URL which was clicked
userAgent	String		The User-Agent header from the clicker's HTTP request

Example Output

<33>1 2016-06-24T21:00:08Z - ProofpointTAP - CLKPER [tapmsg@21139 clickTime="2016-06-24T19:17:44.000Z" messageID="8c6cfedd-3050-4d65-8c09-c5f65c38da81" recipient="bruce.wayne@pharmtech.zz" sender="9facbf452def2d7efc5b5c48cdb837fa@badguy.zz" senderIP="192.0.2.255" campaignId="46e01b8a-c899-404d-bcd9-189bb393d1a7" threatId="2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca" class="MALWARE" threatUrl="https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/61f7622167144dba5e3ae4480eeee78b23d66f7dfed970cfc3d086cc0dabdf50" threatTime="2016-06-24T19:17:46.000Z" url="http://badguy.zz/" userAgent="Mozilla/5.0(WindowsNT6.1;WOW64;rv:27.0)Gecko/20100101Firefox/27.0"]
<38>1 2016-06-24T21:00:08Z - ProofpointTAP - MSGBLK [tapmsg@21139 messageTime="2016-06-24T21:18:38.000Z" messageID="20160624211145.62086.mail@evil.zz" recipient="clark.kent@pharmtech.zz, diana.prince@pharmtech.zz" sender="e99d7ed5580193f36a51f597bc2c0210@evil.zz" senderIP="192.0.2.255" phishScore="46" spamScore="4" QID="r2FNwRHF004109" GUID="c26dbea0-80d5-463b-b93c-4e8b708219ce" threatsInfoMap="[{\"campaignId\" : \"46e01b8a-c899-404d-bcd9-189bb393d1a7\", \"threatId\" : \"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\", \"threatType\" : \"URL\", \"classification\" : \"MALWARE\", \"threatUrl\" : \"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\", \"threatTime\" : \"2016-06-24T21:18:38.000Z\", \"threat\" : \"http://evil.zz/danger/?exploitUser=true\",\"threatStatus\":\"active\"}, {\"campaignId\" : \"46e01b8a-c899-404d-bcd9-189bb393d1a7\", \"threatId\" : \"3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa\", \"threatType\" : \"URL\", \"classification\" : \"MALWARE\", \"threatUrl\" : \"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa\", \"threatTime\" : \"2016-06-24T21:18:07.000Z\", \"threat\" : \"badsite.zz\",\"threatStatus\":\"active\"}\]" malwareScore="100" impostorScore="0" completelyRewritten="true" subject="Please find a totally safe invoice attached." quarantineRule="module.sandbox.threat" quarantineFolder="Attachment Defense" policyRoutes="default_inbound,executives" modulesRun="sandbox,urldefense,spam,pdr" headerFrom="\"A. Badguy\" <badguy@evil.zz>" headerTo="\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>" headerCC="\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>" headerReplyTo="null" toAddresses="clark.kent@pharmtech.zz,diana.prince@pharmtech.zz" ccAddresses="bruce.wayne@university-of-education.zz" fromAddress="badguy@evil.zz" replyToAddress="null" clusterId="pharmtech_hosted" messageParts="[{\"contentType\":\"text/plain\",\"disposition\":\"inline\",\"filename\":\"text.txt\",\"md5\":\"008c5926ca861023c1d2a36653fd88e2\",\"oContentType\":\"text/plain\",\"sandboxStatus\":\"unsupported\",\"sha256\":\"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281\"},{\"contentType\":\"application/pdf\",\"disposition\":\"attached\",\"filename\":\"Invoice for Pharmtech.pdf\",\"md5\":\"5873c7d37608e0d49bcaa6f32b6c731f\",\"oContentType\":\"application/pdf\",\"sandboxStatus\":\"threat\",\"sha256\":\"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\"}]" xmailer="Spambot v2.5"]

Changes from version 1.5 of the SIEM API

There are several breaking changes from the previous major version of the SIEM API. Complete details of the changes are available in the dedicated Changes from the 1.5 SIEM API topic.

Sample Unix Shell Script

This script can be run as a cron job on any Unix OS which supports the bash shell. The service principal and secret must be customized before use. A downloadable version of this script can be found here: Downloadable Shell Script

#!/bin/bash
#title :retrieve-tap-siem-logs.sh
#description :This script retrieves SIEM logs from Proofpoint's Targeted Attack Protection service.
#author :Proofpoint
#Last revised :2020-11-20
#version :1.4
#usage :bash retrieve-tap-siem-logs.sh
############################################################
#Version History
# 1.0 Initial Release
# 1.1 Some old versions of the 'date' command didn't support
# ISO8601-formatted timestamps. Fixed to be friendlier to
# those old versions.
# 1.2 Added threatStatus parameter to query.
# 1.3 Fixed quoting bug which affected customers in timezones with positive UTC offset
# 1.4 Support timestamps on centos6.10 and later
############################################################


#=============USER CONFIGURABLE SETTINGS===================#
# The service principal and secret are used to authenticate to the SIEM API. They are generated on the settings page of the Threat Insight Dashboard.
PRINCIPAL="<principal>"
SECRET="<secret>"

# Determines which API method is used. Valid values are: "all", "issues",
# "messages/blocked", "messages/delivered", "clicks/permitted", and "clicks/blocked"
ACTION="all"

# Determines which format the log is downloaded in. Valid values are "syslog" and "json".
FORMAT="syslog"

# Determines where log file are downloaded. Defaults to the current working directory.
LOGDIR=`pwd`

# Determines which threat statuses to retrieve. Defaults to all three valid threat statuses: active, clear, and falsePositive
STATUSES=(active cleared falsePositive)
#=============END USER CONFIGURABLE SETTINGS===================#

LASTRETRIEVALFILE="$LOGDIR/lastretrieval"
LOGFILESUFFIX="tap-siem.log"
ERRORFILESUFFIX="tap-siem.error"
TMPFILESUFFIX="tap-siem.tmp"
CURRENTTIME_ISO=`date -Iseconds | tr "T" " "`
CURRENTTIME_SECS=`date -d "$CURRENTTIME_ISO" +%s`

STATUS_PARAMETERS=""
STATUSES_LENGTH=${#STATUSES[@]}
for ((i=0; i<$STATUSES_LENGTH - 1; i++)); do
        STATUS_PARAMETERS+="threatStatus=${STATUSES[$i]}&"
done
STATUS_PARAMETERS+="threatStatus=${STATUSES[$STATUSES_LENGTH-1]}"

function escapeTimestamp {
        echo $1 | sed -r "s/\+/%2B/g"
}

function interpretResults {
        local STATUS=$1
        local EXITCODE=$2
        local TIME_ISO=$3
        local TIME_SECS=$4
        if [[ $EXITCODE -eq 0 ]] && [[ $STATUS -eq 200 ]]; then
                echo $TIME_ISO > $LASTRETRIEVALFILE
                mv "$LOGDIR/$TIME_SECS-$TMPFILESUFFIX" "$LOGDIR/$TIME_SECS-$LOGFILESUFFIX"
                echo "Retrieval successful. $LOGDIR/$TIME_SECS-$LOGFILESUFFIX created."
                return 0
        fi
        if [[ $EXITCODE -eq 0 ]] && [[ $STATUS -eq 204 ]]; then
                echo $TIME_ISO > $LASTRETRIEVALFILE
                if [[ -f $LOGDIR/$TIME_SECS-$TMPFILESUFFIX ]]; then
                  rm "$LOGDIR/$TIME_SECS-$TMPFILESUFFIX"
                fi
                echo "Retrieval successful. No new records found."
                return 0
        fi

        mv "$LOGDIR/$TIME_SECS-$TMPFILESUFFIX" "$LOGDIR/$TIME_SECS-$ERRORFILESUFFIX"
        echo "Retrieval unsuccessful. $LOGDIR/$TIME_SECS-$ERRORFILESUFFIX created."
        logger -p user.err "Failed to retrieve TAP SIEM logs. Error in $LOGDIR/$TIME_SECS-$ERRORFILESUFFIX."
        exit 1
}

function retrieveSinceSeconds {
        SECONDS=$1
        STATUS=$(curl -X GET -w %{http_code} -o "$LOGDIR/$CURRENTTIME_SECS-$TMPFILESUFFIX" "https://tap-api-v2.proofpoint.com/v2/siem/$ACTION?$STATUS_PARAMETERS&format=$FORMAT&sinceSeconds=$SECONDS" --user "$PRINCIPAL:$SECRET" -s)
        EXITCODE=$?
        interpretResults $STATUS $EXITCODE "$CURRENTTIME_ISO" "$CURRENTTIME_SECS"
}

function retrieveSinceTime {
        TIME=$(escapeTimestamp $1)
        STATUS=$(curl -X GET -w %{http_code} -o "$LOGDIR/$CURRENTTIME_SECS-$TMPFILESUFFIX" "https://tap-api-v2.proofpoint.com/v2/siem/$ACTION?$STATUS_PARAMETERS&format=$FORMAT&sinceTime=$TIME" --user "$PRINCIPAL:$SECRET" -s)
        EXITCODE=$?
        interpretResults $STATUS $EXITCODE "$CURRENTTIME_ISO" "$CURRENTTIME_SECS"
}

function retrieveInterval {
        START_ISO=$(escapeTimestamp $1)
        END_ISO=$(escapeTimestamp $2)
        END_SECS=$3
        STATUS=$(curl -X GET -w %{http_code} -o "$LOGDIR/$END_SECS-$TMPFILESUFFIX" "https://tap-api-v2.proofpoint.com/v2/siem/$ACTION?$STATUS_PARAMETERS&format=$FORMAT&interval=$START_ISO/$END_ISO" --user "$PRINCIPAL:$SECRET" -s)
        EXITCODE=$?
        interpretResults $STATUS $EXITCODE "$END_ISO" "$END_SECS"
}

if ! [[ -f $LASTRETRIEVALFILE ]]; then
        echo "No interval file found. Retrieving past hour's worth of data."
                retrieveSinceSeconds 3600
        else
                LASTRETRIEVAL_ISO=$(date -f "$LASTRETRIEVALFILE" -Iseconds)
                LASTRETRIEVAL_UNIX=$(date -f "$LASTRETRIEVALFILE" -Iseconds | tr "T" " ")
                LASTRETRIEVAL_SECS=$(date -d "$LASTRETRIEVAL_UNIX" +%s)
                (( DIFF=$CURRENTTIME_SECS - $LASTRETRIEVAL_SECS ))

        if [ $DIFF -lt 60 ]; then
                echo "Last retrieval was $DIFF seconds ago. Minimum amount of between requests is 60 seconds."
                logger -p user.err "Last retrieval was $DIFF seconds ago. Minimum amount of between requests is 60 seconds. Exiting."
                exit 0
        fi

        if [ $DIFF -gt 43200 ]; then
                echo "Last successful retrieval of SIEM logs was $DIFF seconds ago. Maximum amount of time to look back is 43200 seconds (12 hours). Resetting last interval. Information older than 12 hours will not be retrieved."
                logger -p user.warn "Last successful retrieval of SIEM logs was $DIFF seconds ago. Maximum amount of time to look back is 43200 seconds (12 hours). Resetting last interval. Information older than 12 hours will not be retrieved."
                ((LASTRETRIEVAL_SECS=$CURRENTTIME_SECS-43140))
                LASTRETRIEVAL_ISO=`date -d @$LASTRETRIEVAL_SECS -Iseconds`
                (( DIFF= $CURRENTTIME_SECS - $LASTRETRIEVAL_SECS ))
        fi

        if [ $DIFF -gt 3600 ]; then
                echo "Last retrieval was $DIFF seconds ago. Maximum amount of allowable time for one request is 3600 seconds. Will split into several requests."
                START_ISO=$LASTRETRIEVAL_ISO
                START_SECS=$LASTRETRIEVAL_SECS
                while [ $DIFF -gt 3600 ]; do
                        ((END_SECS=$START_SECS+3600))
                        END_ISO=`date -d @$END_SECS -Iseconds`
                        (( DIFF=$CURRENTTIME_SECS - $END_SECS ))
                        retrieveInterval $START_ISO $END_ISO $END_SECS
                        START_SECS=$END_SECS
                        START_ISO=$END_ISO
                done
                LASTRETRIEVAL_ISO=$END_ISO
        fi

        if [ $DIFF -le 3600 ]; then
                retrieveSinceTime $LASTRETRIEVAL_ISO
        fi
fi

Overview

API Features

General Service Notes

Security

Standard Responses

Throttle Limits

Available Endpoints

/v2/siem/clicks/blocked

/v2/siem/clicks/permitted

/v2/siem/messages/blocked

/v2/siem/messages/delivered

/v2/siem/issues

/v2/siem/all

Required Parameters

interval

sinceSeconds

sinceTime

Optional Parameters

format

threatType

threatStatus

Example Commands in Curl

Note About Event Time

Results

JSON

Structure

Message Events

Click Events

Example Output

syslog

Structure

Message Event Data

Click Event Data

Example Output

Changes from version 1.5 of the SIEM API

Sample Unix Shell Script