Skip to main content
Contact Us
My preferencesSign out
Proofpoint, Inc.

SIEM API

  1. Last updated
  2. Save as PDF

The information and data accessible via this API contain Proofpoint proprietary, confidential, and/or trade secret information. Sharing or providing the information to another party without Proofpoint's express written consent is prohibited. Please review the updated Terms of Use for Proofpoint APIs. The TAP API Terms of Use can be found online at API Terms of Use | Proofpoint US.

Overview

Security Information and Event Management (SIEM) solutions are used by many organizations to identify and correlate various security events occurring in their point products. Examples of SIEM products include HP's ArcSight, IBM's QRadar, and Splunk. 

The API allows integration with these solutions by giving administrators the ability to periodically download detailed information about several types of TAP events in a SIEM-compatible, vendor-neutral format. Currently, the following event types are exposed:

  1. Blocked or permitted clicks to threats recognized by URL Defense
  2. Blocked or delivered messages that contain threats recognized by URL Defense or Attachment Defense

API Features

General Service Notes

  1. All timestamps in the returned events are in UTC.
  2. The API is designed to support different SIEM-compatible formats: Syslog and JSON.
  3. A maximum of one hour of data can be requested in a single transaction.
  4. No paging support is available; all the applicable events in the requested time period will be returned in the log.
  5. The results provided by this API may not be in any logical order.
  6. Requests to the service may be throttled to prevent abuse.
  7. If results cannot be obtained within a timeout period, the service will return an error.
  8. The maximum time into the past that can be queried is 7 days with a maximum fetch time of 1 hour.

Security

Each request must use:

  • SSL.
  • service credentials to authenticate to the API.
  • the HTTP Basic Authorization method.
  • the HTTP GET method.

Standard Responses

Requests to the endpoints can produce a response with a variety of HTTP status codes. The following table describes the scenarios in which these codes can be produced.

Code Message Scenarios
200 Success At least one record matching the specified criteria was found and returned in the response body. In the case of a JSON format, the structure is always returned, even if empty.

204

 No Content

Syslog format only: If no records matching the specified criteria were found, a status code of 204 will be returned with empty content.

400

 Bad Request

The request is missing a mandatory "request" parameter, a parameter contains data which is incorrectly formatted, or the API doesn't have enough information to determine the identity of the customer.

401

 Unauthorized

There is no authorization information included in the request, the authorization information is incorrect, or the user is not authorized.

403

 Forbidden

The user is authenticated for the service but is not authorized to access data for the given customer.

429

 Too Many Requests

The user has made too many requests over the past 24 hours and has been throttled.

500

 Internal Server Error

The service has encountered an unexpected situation and is unable to give a better response to the request.

Throttle Limits

These endpoints provide methods to fetch information about click and message events for a given time period. The number of queries connected to this resource are limited by a simple, rolling 24-hour throttle. Once exceeded, the API will start returning 429 HTTP status codes until 24 hours past the oldest request has elapsed. Requests to the clicks/permitted API and requests to other APIs are throttled into different pools. This allows more frequent queries to the clicks/permitted API.

Endpoint

Max Number of Requests
clicks/permitted 1800 per 24 hours

all

issues

clicks/blocked

messages/delivered

messages/blocked

 1800 per 24 hours

 

Available Endpoints

All endpoints are available on the tap-api-v2.proofpoint.com host—for example, https://tap-api-v2.proofpoint.com/v2...clicks/blocked.

/v2/siem/clicks/blocked

Fetch events for clicks to malicious URLs blocked in the specified time period

/v2/siem/clicks/permitted

Fetch events for clicks to malicious URLs permitted in the specified time period

/v2/siem/messages/blocked

Fetch events for messages blocked in the specified time period which contained a known threat

/v2/siem/messages/delivered

Fetch events for messages delivered in the specified time period which contained a known threat

/v2/siem/issues

Fetch events for clicks to malicious URLs permitted and messages delivered containing a known threat within the specified time period. (It is a combination of /v2/siem/clicks/permitted and /v2/siem/messages/delivered)

/v2/siem/all

Fetch events for all clicks and messages relating to known threats within the specified time period

Required Parameters

One of the following three query parameters describing the desired time range for the data must be supplied with each request:

interval

A string containing an ISO8601-formatted interval. If this interval overlaps with previous requests for data, records from the previous request may be duplicated. The minimum interval is thirty seconds. The maximum interval is one hour.

Examples:

  • 2016-05-01T12:00:00Z/2016-05-01T13:00:00Z - an hour interval, beginning at noon UTC on 05-01-2016
  • PT30M/2016-05-01T12:30:00Z - the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC
  • 2016-05-01T05:00:00-0700/PT30M - the same interval as above, but using -0700 as the time zone
sinceSeconds 

An integer representing a time window in seconds from the current API server time. The start of the window is the current API server time, rounded to the nearest minute, less the number of seconds provided. The end of the window is the current API server time rounded to the nearest minute. If JSON output is selected, the end time is included in the returned result.

sinceTime

A string containing an ISO8601 date. It represents the start of the data retrieval period. The end of the period is determined by current API server time rounded to the nearest minute. If JSON output is selected, the end time is included in the returned result.

Optional Parameters

One or more of these parameters may also be provided:

format

A string specifying the format in which data is returned. If no format is specified, syslog will be used as the default. The following values are accepted:

  • syslog
  • JSON
threatType

A string specifying which threat type will be returned in the data. If no value is specified, all threat types are returned. The following values are accepted:

  • url
  • attachment
  • messageText
threatStatus

A string specifying which threat statuses will be returned in the data. If no value is specified, active and cleared threats are returned. The following values are accepted:

  • active
  • cleared
  • falsePositive

Example Commands in Curl

The following commands assume that principal and secret are defined environment variables. They correspond to the service principal and secret that was created on the Settings page.

curl "https://tap-api-v2.proofpoint.com/v2/siem/issues?format=json&sinceTime=2016-05-01T12:00:00Z" --user "$PRINCIPAL:$SECRET" -s

Retrieves events from noon on 05/01/2016 to the present. Returned events are limited to just permitted clicks and delivered messages with known threats. Output is in the JSON format.

curl "https://tap-api-v2.proofpoint.com/v2/siem/all?format=syslog&sinceSeconds=3600" --user "$PRINCIPAL:$SECRET" -s

Retrieves events to the present, starting 3600 seconds before the query time. All events are returned. Output is in the syslog Format.

curl "https://tap-api-v2.proofpoint.com/v2/siem/clicks/permitted?format=syslog&interval=PT30M/2016-05-01T12:30:00Z" --user "$PRINCIPAL:$SECRET" -s

Retrieves events from the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC. Only permitted clicks are returned. Output is in the syslog format.

curl "https://tap-api-v2.proofpoint.com/v2/siem/all?format=json&interval=PT30M/2016-05-01T12:30:00Z&threatStatus=falsePositive&threatStatus=active&threatStatus=cleared" --user "$PRINCIPAL:$SECRET" -s

Retrieves events from the thirty minutes beginning at noon UTC on 05-01-2016 and ending at 12:30pm UTC. All events are returned. False positives are included in the output. Output is in the JSON format.

Note About Event Time

Event Time in SIEM is the time the event was ingested into the SIEM. The time range used in the query parameters controls which events the SIEM API returns based on the time that the event was created, not the time the event occurred. The time an event is created is always after either of these two times:

  • the time that the message was sent or the time click occurred (messageTime or clickTime)
  • the time that the threat referenced by the message or click was recognized and condemned by Proofpoint (threatsInfoMap/threatTime)

In other words, a request using the sinceSeconds=3600 parameter will retrieve all events which have been created in the last hour. It is possible that the events returned from that interval reference messages or clicks which were first observed more than one hour ago—perhaps even several days ago.

Results

JSON 

If the JSON output is used, the following structure will always be produced, even if there are no events inside any individual (or all) event arrays.

Structure

Fields

Type

Restrictions

Description

queryEndTime

DateTime

ISO8601 Timestamp

The time at which the period queried for data ended
messagesDelivered Array Message Events An array containing all messages with threats which were delivered by PPS
messagesBlocked Array Message Events An array containing all messages with threats which were quarantined by PPS
clicksPermitted Array Click Events An array containing all clicks to URL threats which were permitted
clicksBlocked Array Click Events An array containing all clicks to URL threats which were blocked
Message Events

Field

Content

Restrictions

Description
ccAddresses Array of strings   A list of email addresses contained within the CC: header, excluding friendly names.
clusterId String   The name of the PPS cluster which processed the message.
completelyRewritten String
  • true
  • false
  • na
 The rewrite status of the message. If value is 'true', all instances of URL threats within the message were successfully rewritten. If the value is 'false', at least one instance of the a threat URL was not rewritten. If the value is 'na', the message did not contain any URL-based threats.
fromAddress String   The email address contained in the From: header, excluding friendly name.
GUID String   The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
headerFrom String   The full content of the From: header, including any friendly name.
headerReplyTo String    If present, the full content of the Reply-To: header, including any friendly names.
impostorScore int 0-100 The impostor score of the message. Higher scores indicate higher certainty.
malwareScore int 0-100 The malware score of the message. Higher scores indicate higher certainty.

messageID

String

  

Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique.
messageParts Array of JSON Structures   An array of structures which contain details about parts of the message, including both message bodies and attachments.
messageParts/contentType String   The true, detected Content-Type of the messagePart. This may differ from the oContentType value.
messageParts/disposition String
  • inline
  • attached
 If the value is "inline," the messagePart is a message body. If the value is "attached," the messagePart is an attachment.
messageParts/filename String   The filename of the messagePart.
messageParts/md5 String   The MD5 hash of the messagePart contents.
messageParts/oContentType String   The declared Content-Type of the messagePart.
messageParts/sandboxStatus String
  • unsupported
  • threat
  • clean
  • prefilter
  • uploaded
  • inprogress
  • uploaddisabled
 The verdict returned by the sandbox during the scanning process. If the value is "unsupported", the messagePart is not supported by Attachment Defense and was not scanned. If the value is "clean", the sandbox returned a clean verdict. If the value is "threat", the sandbox returned a malicious verdict. If the value is "prefilter", the messagePart contained no active content, and was therefore not sent to the sandboxing service. If the value is "uploaded," the message was uploaded by PPS to the sandboxing service, but did not yet have a verdict at the time the message was processed. If the value is "inprogress," the attachment had been uploaded and was awaiting scanning at the time the message was processed. If the verdict is "uploaddisabled," the attachment was eligible for scanning, but was not uploaded because of PPS policy.
messageParts/sha256 String   The SHA256 hash of the messagePart contents.
messageSize Integer   The size in bytes of the message, including headers and attachments.

messageTime

DateTime

ISO8601 Timestamp

When the message was delivered to the user or quarantined by PPS
modulesRun Array of strings   The list of PPS modules which processed the message.

phishScore

int

0-100

The phish score of the message. Higher scores indicate higher certainty.
policyRoutes Array of strings   The policy routes that the message matched during processing by PPS.
QID String   The queue ID of the message within PPS. It can be used to identify the message in PPS and is not unique.
quarantineFolder String   The name of the folder which contains the quarantined message. This appears only for messagesBlocked.
quarantineRule String   The name of the rule which quarantined the message. This appears only for messagesBlocked events.

recipient

String

Email Address

An array containing the email addresses of the SMTP (envelope) recipients
replyToAddress String   The email address contained in the Reply-To: header, excluding friendly name.

sender

String

Email Address

The email address of the SMTP (envelope) sender. The user-part is hashed. The domain-part is cleartext.

senderIP

String

IP Address

The IP address of the sender.

spamScore

int

0-100

The spam score of the message. Higher scores indicate higher certainty.
subject String   The subject line of the message, if available.
threatsInfoMap Array of JSON Structures   An array of structures which contain details about detected threats within the message. There may be more than one threat per message.
threatsInfoMap/detectionType String Nullable New field detectionType and ThreatInfo added to response
threatsInfoMap/campaignId String   An identifier for the campaign of which the threat is a member, if available at the time of the query. Threats can be linked to campaigns even after these events are retrieved.
threatsInfoMap/classification String
  • Malware
  • Phish
  • Spam
  • Impostor (for BEC/Message Text threats)
  • TOAD (Telephone-Oriented Attack Delivery)
 The category of threat found in the message

threatsInfoMap/threat

String
  • SHA256
  • URL
  • email address
 The artifact which was condemned by Proofpoint. The malicious URL, hash of the attachment threat, or email address of the impostor sender.

threatsInfoMap/threatId

String

   The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.
threatsInfoMap/threatStatus String
  • active
  • falsepositive
  • cleared
 The current state of the threat.

threatsInfoMap/threatTime

DateTime

ISO8601 Timestamp

Proofpoint assigned the threatStatus at this time.

threatsInfoMap/threatType

String
  • Attachment
  • URL
  • Message

Whether the threat was an attachment, URL, or message type.

threatsInfoMap/threatUrl

String

 URL A link to the entry about the threat on the TAP Dashboard
threatsInfoMap/actors Array of JSON Structures   An array of structures which contain details about the actors associated with a threat
toAddresses Array of strings   A list of email addresses contained within the To: header, excluding friendly names.
xmailer String   The content of the X-Mailer: header, if present.
Click Events

Field 

Content 

Restrictions 

Description 
campaignId String   An identifier for the campaign of which the threat is a member, if available at the time of the query. Threats can be linked to campaigns even after these events are retrieved.
classification String
  • Malware
  • Phish
  • Spam
 The threat category of the malicious URL
clickIP String IP Address The external IP address of the user who clicked on the link. If the user is behind a firewall performing network address translation, the IP address of the firewall will be shown.

clickTime

DateTime

ISO8601 Timestamp

The time the user clicked on the URL

GUID

String

 

The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
id String   The unique id of the click.

recipient

String

Email Address

The email address of the recipient

sender

String

Email Address

The email address of the sender. The user-part is hashed. The domain-part is cleartext.

senderIP

String

IP Address

The IP address of the sender

threatID

String

 

The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints. 

threatTime

DateTime

ISO8601 Timestamp

Proofpoint identified the URL as a threat at this time.

threatURL

String

URL

A link to the entry on the TAP Dashboard for the particular threat
threatStatus String
  • active
  • falsepositive
  • cleared
 The current state of the threat.

url

String

URL

The malicious URL which was clicked
userAgent String   The User-Agent header from the clicker's HTTP request
Example Output
{
   "clicksPermitted":[
      {
         "campaignId":"46e01b8a-c899-404d-bcd9-189bb393d1a7",
         "classification":"MALWARE",
         "clickIP":"192.0.2.1",
         "clickTime":"2016-06-24T19:17:44.000Z",
         "GUID": "b27dbea0-87d5-463b-b93c-4e8b708289ce",
         "id": "8c8b4895-a277-449f-r797-547e3c89b25a",
         "messageID":"8c6cfedd-3050-4d65-8c09-c5f65c38da81",
         "recipient":"bruce.wayne@pharmtech.zz",
         "sender":"9facbf452def2d7efc5b5c48cdb837fa@badguy.zz",
         "senderIP":"192.0.2.255",
         "threatID":"61f7622167144dba5e3ae4480eeee78b23d66f7dfed970cfc3d086cc0dabdf50",
         "threatTime":"2016-06-24T19:17:46.000Z",
         "threatURL":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/61f7622167144dba5e3ae4480eeee78b23d66f7dfed970cfc3d086cc0dabdf50",
         "threatStatus": "active",
         "url":"http://badguy.zz/",
         "userAgent":"Mozilla/5.0(WindowsNT6.1;WOW64;rv:27.0)Gecko/20100101Firefox/27.0"
      }
   ],
   "messagesBlocked":[
      {
         "GUID":"c26dbea0-80d5-463b-b93c-4e8b708219ce",
         "QID":"r2FNwRHF004109",
         "ccAddresses":[
            "bruce.wayne@university-of-education.zz"
         ],
         "clusterId":"pharmtech_hosted",
         "completelyRewritten":"true",
         "fromAddress":"badguy@evil.zz",
         "headerCC":"\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>",
         "headerFrom":"\"A. Badguy\" <badguy@evil.zz>",
         "headerReplyTo":null,
         "headerTo":"\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>",
         "impostorScore":0,
         "malwareScore":100,
         "messageID":"20160624211145.62086.mail@evil.zz",
         "messageParts":[
            {
               "contentType":"text/plain",
               "disposition":"inline",
               "filename":"text.txt",
               "md5":"008c5926ca861023c1d2a36653fd88e2",
               "oContentType":"text/plain",
               "sandboxStatus":"unsupported",
               "sha256":"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281"
            },
            {
               "contentType":"application/pdf",
               "disposition":"attached",
               "filename":"Invoice for Pharmtech.pdf",
               "md5":"5873c7d37608e0d49bcaa6f32b6c731f",
               "oContentType":"application/pdf",
               "sandboxStatus":"threat",
               "sha256":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
            }
         ],
         "messageTime":"2016-06-24T21:18:38.000Z",
         "modulesRun":[
            "pdr",
            "sandbox",
            "spam",
            "urldefense"
         ],
         "phishScore":46,
         "policyRoutes":[
            "default_inbound",
            "executives"
         ],
         "quarantineFolder":"Attachment Defense",
         "quarantineRule":"module.sandbox.threat",
         "recipient":[
            "clark.kent@pharmtech.zz",
            "diana.prince@pharmtech.zz"
         ],
         "replyToAddress":null,
         "sender":"e99d7ed5580193f36a51f597bc2c0210@evil.zz",
         "senderIP":"192.0.2.255",
         "spamScore":4,
         "subject":"Please find a totally safe invoice attached.",
         "threatsInfoMap": [
            {  
               "campaignId": "46e01b8a-c899-404d-bcd9-189bb393d1a7",
               "classification":"MALWARE",
               "threat":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
               "threatId":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
               "threatStatus":"active",
               "threatTime":"2016-06-24T21:18:38.000Z",
               "threatType":"ATTACHMENT",
               "threatUrl":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
               "actors": [
                   {    
                        "id": "000090ac-71c4-4060-9154-c590664b6739",
                        "name": "dash_actor_05cbc5",
                        "type": "ACTOR"
                   }
               ],
            },
            {               
               "threatID": "d57d32f7ad46db1c233eb586da7fa39bafba67ec0f8d1a4eae03b3df65e2c60e",
               "threatStatus": "active",
               "classification": "malware",
               "detectionType": "NONE",
               "threatUrl": "https://tap-dashboard-staging.lab.ppops.net/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/email/d57d32f7ad46db1c233eb586da7fa39bafba67ec0f8d1a4eae03b3df65e2c60e",
               "threatTime": "2024-11-20T12:01:07.000Z",
               "threat": "http://rohan.org/willis",
               "campaignID": null,
               "actors": [
                   {   
                        "id": "000090ac-71c4-4060-9154-c590664b6739",
                        "name": "dash_actor_05cbc5",
                        "type": "ACTOR"
                   }
               ],
               "threatType": "url"
            }  
         ],

syslog

Individual events are CRLF-delimited. Events are produced in the syslog format, as described by RFC 5424. All data is contained within the structured-data field. The freeform MSG field is blank. 

Structure

Event Type (MSGID)

 Priority Value SD-ID

Restrictions

Description
MSGDLV <36> (Warning) tapmsg@21139 Message Events A message containing a threat was delivered by PPS.
MSGBLK <38> (Informational) tapmsg@21139 Message Events A message containing a threat was quarantined by PPS.
CLKPER <33> (Alert) tapclk@21139 Click Events A click to a threat was permitted.
CLKBLK <38> (Informational) tapclk@21139 Click Events A click to a threat was blocked.
Message Event Data

Field

Content

Restrictions

Description
ccAddresses String Comma-delimited A list of email addresses contained within the CC: header, excluding friendly names.
clusterId String   The name of the PPS cluster which processed the message.
completelyRewritten String
  • true
  • false
  • na
 The rewrite status of the message. If value is 'true', all instances of URL threats within the message were successfully rewritten. If the value is 'false', at least one instance of the a threat URL was not rewritten. If the value is 'na', the message did not contain any URL-based threats.
fromAddress String   The email address contained in the From: header, excluding friendly name.
GUID String   The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.
headerFrom String   The full content of the From: header, including any friendly name.
headerReplyTo String    If present, the full content of the Reply-To: header, including any friendly names.
impostorScore int 0-100 The impostor score of the message. Higher scores indicate higher certainty.
malwareScore int 0-100 The malware score of the message. Higher scores indicate higher certainty.

messageID

String

  

Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique.
messageParts Map JSON Structure An array of structures which contain details about parts of the message, including both message bodies and attachments. The structure is exactly the same as the above.
messageSize Integer   The size in bytes of the message, including headers and attachments.

messageTime

DateTime

ISO8601 Timestamp

When the message was delivered to the user or quarantined by PPS
modulesRun String Comma-delimited The list of PPS modules which processed the message.

phishScore

int

0-100

The phish score of the message. Higher scores indicate higher certainty.
policyRoutes Strings Comma-delimited The policy routes that the message matched during processing by PPS.
QID String   The queue ID of the message within PPS. It can be used to identify the message in PPS and is not unique.
quarantineFolder String   The name of the folder which contains the quarantined message. This appears only for messagesBlocked.
quarantineRule String   The name of the rule which quarantined the message. This appears only for messagesBlocked events.

recipient

String

Email Address

An array containing the email addresses of the recipients
replyToAddress String   The email address contained in the Reply-To: header, excluding friendly name.

sender

String

Email Address

The email address of the sender. The user-part is hashed. The domain-part is cleartext.

senderIP

String

IP Address

The IP Address of the sender

spamScore

int

0-100

The spam score of the message. Higher scores indicate higher certainty.
subject String   The subject line of the message, if available.
threatsInfoMap Map JSON Structure A string containing a JSON structure with details about detected threats within the message. There may be more than one threat per message. The threatsInfoMap structure is exactly the same as the JSON output above.
toAddresses String Comma-delimited A list of email addresses contained within the To: header, excluding friendly names.
xmailer String   The content of the X-Mailer: header, if present.
Click Event Data

Field 

Content 

Restrictions 

Description 
campaignId String   An identifier for the campaign of which the threat is a member, if available at the time of the query. Threats can be linked to campaigns even after these events are retrieved.
class String
  • Malware
  • Phish
  • Spam
 The threat category of the malicious URL
clickIP String IP Address The external IP address of the user who clicked on the link. If the user is behind a firewall performing network address translation, the IP address of the firewall will be shown.

clickTime

DateTime

ISO8601 Timestamp

The time the user clicked on the URL
messageID String   Message-ID extracted from the headers of the email message. It can be used to look up the associated message in PPS and is not unique.

GUID

String

 

The ID of the message within PPS. It can be used to identify the message in PPS and is guaranteed to be unique.

recipient

String

Email Address

The email address of the recipient

sender

String

Email Address

The email address of the sender. The user-part is hashed. The domain-part is cleartext.

senderIP

String

IP Address

The IP address of the sender

threatID

String

 

The unique identifier associated with this threat. It can be used to query the forensics and campaign endpoints.

threatTime

DateTime

ISO8601 Timestamp

Proofpoint identified the URL as a threat at this time.

threatURL

String

URL

A link to the entry on the TAP Dashboard for the particular threat.

url

String

URL

The malicious URL which was clicked
userAgent String   The User-Agent header from the clicker's HTTP request
Example Output
<33>1 2016-06-24T21:00:08Z - ProofpointTAP - CLKPER [tapmsg@21139 clickTime="2016-06-24T19:17:44.000Z" messageID="8c6cfedd-3050-4d65-8c09-c5f65c38da81" recipient="bruce.wayne@pharmtech.zz" sender="9facbf452def2d7efc5b5c48cdb837fa@badguy.zz" senderIP="192.0.2.255" campaignId="46e01b8a-c899-404d-bcd9-189bb393d1a7" threatId="2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca" class="MALWARE" threatUrl="https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/61f7622167144dba5e3ae4480eeee78b23d66f7dfed970cfc3d086cc0dabdf50" threatTime="2016-06-24T19:17:46.000Z" url="http://badguy.zz/" userAgent="Mozilla/5.0(WindowsNT6.1;WOW64;rv:27.0)Gecko/20100101Firefox/27.0"]
<38>1 2016-06-24T21:00:08Z - ProofpointTAP - MSGBLK [tapmsg@21139 messageTime="2016-06-24T21:18:38.000Z" messageID="20160624211145.62086.mail@evil.zz" recipient="clark.kent@pharmtech.zz, diana.prince@pharmtech.zz" sender="e99d7ed5580193f36a51f597bc2c0210@evil.zz" senderIP="192.0.2.255" phishScore="46" spamScore="4" QID="r2FNwRHF004109" GUID="c26dbea0-80d5-463b-b93c-4e8b708219ce" threatsInfoMap="[{\"campaignId\" : \"46e01b8a-c899-404d-bcd9-189bb393d1a7\", \"threatId\" : \"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\", \"threatType\" : \"URL\", \"classification\" : \"MALWARE\", \"threatUrl\" : \"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\", \"threatTime\" : \"2016-06-24T21:18:38.000Z\", \"threat\" : \"http://evil.zz/danger/?exploitUser=true\",\"threatStatus\":\"active\"}, {\"campaignId\" : \"46e01b8a-c899-404d-bcd9-189bb393d1a7\", \"threatId\" : \"3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa\", \"threatType\" : \"URL\", \"classification\" : \"MALWARE\", \"threatUrl\" : \"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa\", \"threatTime\" : \"2016-06-24T21:18:07.000Z\", \"threat\" : \"badsite.zz\",\"threatStatus\":\"active\"}\]" malwareScore="100" impostorScore="0" completelyRewritten="true" subject="Please find a totally safe invoice attached." quarantineRule="module.sandbox.threat" quarantineFolder="Attachment Defense" policyRoutes="default_inbound,executives" modulesRun="sandbox,urldefense,spam,pdr" headerFrom="\"A. Badguy\" <badguy@evil.zz>" headerTo="\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>" headerCC="\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>" headerReplyTo="null" toAddresses="clark.kent@pharmtech.zz,diana.prince@pharmtech.zz" ccAddresses="bruce.wayne@university-of-education.zz" fromAddress="badguy@evil.zz" replyToAddress="null" clusterId="pharmtech_hosted" messageParts="[{\"contentType\":\"text/plain\",\"disposition\":\"inline\",\"filename\":\"text.txt\",\"md5\":\"008c5926ca861023c1d2a36653fd88e2\",\"oContentType\":\"text/plain\",\"sandboxStatus\":\"unsupported\",\"sha256\":\"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281\"},{\"contentType\":\"application/pdf\",\"disposition\":\"attached\",\"filename\":\"Invoice for Pharmtech.pdf\",\"md5\":\"5873c7d37608e0d49bcaa6f32b6c731f\",\"oContentType\":\"application/pdf\",\"sandboxStatus\":\"threat\",\"sha256\":\"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\"}]" xmailer="Spambot v2.5"]

 

Changes from version 1.5 of the SIEM API

There are several breaking changes from the previous major version of the SIEM API. Complete details of the changes are available in the dedicated Changes from the 1.5 SIEM API topic.

Sample Unix Shell Script

This script can be run as a cron job on any Unix OS which supports the bash shell. The service principal and secret must be customized before use. A downloadable version of this script can be found here: Downloadable Shell Script

#!/bin/bash
#title :retrieve-tap-siem-logs.sh
#description :This script retrieves SIEM logs from Proofpoint's Targeted Attack Protection service.
#author :Proofpoint
#Last revised :2020-11-20
#version :1.4
#usage :bash retrieve-tap-siem-logs.sh
############################################################
#Version History
# 1.0 Initial Release
# 1.1 Some old versions of the 'date' command didn't support
# ISO8601-formatted timestamps. Fixed to be friendlier to
# those old versions.
# 1.2 Added threatStatus parameter to query.
# 1.3 Fixed quoting bug which affected customers in timezones with positive UTC offset
# 1.4 Support timestamps on centos6.10 and later
############################################################


#=============USER CONFIGURABLE SETTINGS===================#
# The service principal and secret are used to authenticate to the SIEM API. They are generated on the settings page of the Threat Insight Dashboard.
PRINCIPAL="<principal>"
SECRET="<secret>"

# Determines which API method is used. Valid values are: "all", "issues",
# "messages/blocked", "messages/delivered", "clicks/permitted", and "clicks/blocked"
ACTION="all"

# Determines which format the log is downloaded in. Valid values are "syslog" and "json".
FORMAT="syslog"

# Determines where log file are downloaded. Defaults to the current working directory.
LOGDIR=`pwd`

# Determines which threat statuses to retrieve. Defaults to all three valid threat statuses: active, clear, and falsePositive
STATUSES=(active cleared falsePositive)
#=============END USER CONFIGURABLE SETTINGS===================#

LASTRETRIEVALFILE="$LOGDIR/lastretrieval"
LOGFILESUFFIX="tap-siem.log"
ERRORFILESUFFIX="tap-siem.error"
TMPFILESUFFIX="tap-siem.tmp"
CURRENTTIME_ISO=`date -Iseconds | tr "T" " "`
CURRENTTIME_SECS=`date -d "$CURRENTTIME_ISO" +%s`

STATUS_PARAMETERS=""
STATUSES_LENGTH=${#STATUSES[@]}
for ((i=0; i<$STATUSES_LENGTH - 1; i++)); do
        STATUS_PARAMETERS+="threatStatus=${STATUSES[$i]}&"
done
STATUS_PARAMETERS+="threatStatus=${STATUSES[$STATUSES_LENGTH-1]}"

function escapeTimestamp {
        echo $1 | sed -r "s/\+/%2B/g"
}

function interpretResults {
        local STATUS=$1
        local EXITCODE=$2
        local TIME_ISO=$3
        local TIME_SECS=$4
        if [[ $EXITCODE -eq 0 ]] && [[ $STATUS -eq 200 ]]; then
                echo $TIME_ISO > $LASTRETRIEVALFILE
                mv "$LOGDIR/$TIME_SECS-$TMPFILESUFFIX" "$LOGDIR/$TIME_SECS-$LOGFILESUFFIX"
                echo "Retrieval successful. $LOGDIR/$TIME_SECS-$LOGFILESUFFIX created."
                return 0
        fi
        if [[ $EXITCODE -eq 0 ]] && [[ $STATUS -eq 204 ]]; then
                echo $TIME_ISO > $LASTRETRIEVALFILE
                if [[ -f $LOGDIR/$TIME_SECS-$TMPFILESUFFIX ]]; then
                  rm "$LOGDIR/$TIME_SECS-$TMPFILESUFFIX"
                fi
                echo "Retrieval successful. No new records found."
                return 0
        fi

        mv "$LOGDIR/$TIME_SECS-$TMPFILESUFFIX" "$LOGDIR/$TIME_SECS-$ERRORFILESUFFIX"
        echo "Retrieval unsuccessful. $LOGDIR/$TIME_SECS-$ERRORFILESUFFIX created."
        logger -p user.err "Failed to retrieve TAP SIEM logs. Error in $LOGDIR/$TIME_SECS-$ERRORFILESUFFIX."
        exit 1
}

function retrieveSinceSeconds {
        SECONDS=$1
        STATUS=$(curl -X GET -w %{http_code} -o "$LOGDIR/$CURRENTTIME_SECS-$TMPFILESUFFIX" "https://tap-api-v2.proofpoint.com/v2/siem/$ACTION?$STATUS_PARAMETERS&format=$FORMAT&sinceSeconds=$SECONDS" --user "$PRINCIPAL:$SECRET" -s)
        EXITCODE=$?
        interpretResults $STATUS $EXITCODE "$CURRENTTIME_ISO" "$CURRENTTIME_SECS"
}

function retrieveSinceTime {
        TIME=$(escapeTimestamp $1)
        STATUS=$(curl -X GET -w %{http_code} -o "$LOGDIR/$CURRENTTIME_SECS-$TMPFILESUFFIX" "https://tap-api-v2.proofpoint.com/v2/siem/$ACTION?$STATUS_PARAMETERS&format=$FORMAT&sinceTime=$TIME" --user "$PRINCIPAL:$SECRET" -s)
        EXITCODE=$?
        interpretResults $STATUS $EXITCODE "$CURRENTTIME_ISO" "$CURRENTTIME_SECS"
}

function retrieveInterval {
        START_ISO=$(escapeTimestamp $1)
        END_ISO=$(escapeTimestamp $2)
        END_SECS=$3
        STATUS=$(curl -X GET -w %{http_code} -o "$LOGDIR/$END_SECS-$TMPFILESUFFIX" "https://tap-api-v2.proofpoint.com/v2/siem/$ACTION?$STATUS_PARAMETERS&format=$FORMAT&interval=$START_ISO/$END_ISO" --user "$PRINCIPAL:$SECRET" -s)
        EXITCODE=$?
        interpretResults $STATUS $EXITCODE "$END_ISO" "$END_SECS"
}

if ! [[ -f $LASTRETRIEVALFILE ]]; then
        echo "No interval file found. Retrieving past hour's worth of data."
                retrieveSinceSeconds 3600
        else
                LASTRETRIEVAL_ISO=$(date -f "$LASTRETRIEVALFILE" -Iseconds)
                LASTRETRIEVAL_UNIX=$(date -f "$LASTRETRIEVALFILE" -Iseconds | tr "T" " ")
                LASTRETRIEVAL_SECS=$(date -d "$LASTRETRIEVAL_UNIX" +%s)
                (( DIFF=$CURRENTTIME_SECS - $LASTRETRIEVAL_SECS ))

        if [ $DIFF -lt 60 ]; then
                echo "Last retrieval was $DIFF seconds ago. Minimum amount of between requests is 60 seconds."
                logger -p user.err "Last retrieval was $DIFF seconds ago. Minimum amount of between requests is 60 seconds. Exiting."
                exit 0
        fi

        if [ $DIFF -gt 43200 ]; then
                echo "Last successful retrieval of SIEM logs was $DIFF seconds ago. Maximum amount of time to look back is 43200 seconds (12 hours). Resetting last interval. Information older than 12 hours will not be retrieved."
                logger -p user.warn "Last successful retrieval of SIEM logs was $DIFF seconds ago. Maximum amount of time to look back is 43200 seconds (12 hours). Resetting last interval. Information older than 12 hours will not be retrieved."
                ((LASTRETRIEVAL_SECS=$CURRENTTIME_SECS-43140))
                LASTRETRIEVAL_ISO=`date -d @$LASTRETRIEVAL_SECS -Iseconds`
                (( DIFF= $CURRENTTIME_SECS - $LASTRETRIEVAL_SECS ))
        fi

        if [ $DIFF -gt 3600 ]; then
                echo "Last retrieval was $DIFF seconds ago. Maximum amount of allowable time for one request is 3600 seconds. Will split into several requests."
                START_ISO=$LASTRETRIEVAL_ISO
                START_SECS=$LASTRETRIEVAL_SECS
                while [ $DIFF -gt 3600 ]; do
                        ((END_SECS=$START_SECS+3600))
                        END_ISO=`date -d @$END_SECS -Iseconds`
                        (( DIFF=$CURRENTTIME_SECS - $END_SECS ))
                        retrieveInterval $START_ISO $END_ISO $END_SECS
                        START_SECS=$END_SECS
                        START_ISO=$END_ISO
                done
                LASTRETRIEVAL_ISO=$END_ISO
        fi

        if [ $DIFF -le 3600 ]; then
                retrieveSinceTime $LASTRETRIEVAL_ISO
        fi
fi