Pulling Non Subscribed Pulses

[GTownson]

I have only subscribed to Alienvault, but the TI plugin seems to be pulling Pulses that I haven't subscribed to. This is causing issues as I have about 50k threats detected as the plugin has pulled the pulse "dont subscribe". This pulse contains only one indicator which is 8.8.8.8, so when anyone in our organisation tries to query Google's DNS we are getting threat indicated.

I am not fully sure that this is an issue in the plugin, or some kind of configuration issue on our end, but some help would be greatly appreciated. Other than this issue, what a great plugin, keep up the good work!

[jasonkeller]

I can verify this @jalogisch , and better I can point you and @dennisoelkers at exactly what's wrong. Not to demean anyone's efforts on it (and it is appreciated), but until we fix the below issues it will be absolutely useless to anyone that attempts to use it due to the significant amounts of noise generated.

From https://github.com/Graylog2/graylog-plugin-threatintel/blob/master/src/main/java/org/graylog/plugins/threatintel/functions/otx/OTXLookupResult.java ...

public static OTXLookupResult buildFromIntel(OTXIntel intel) {
        if(intel.getPulseCount() > 0) {

The API doesn't prune pulse responses by API key (a lookup will return all pulses that match that indicator regardless of subscription, which is a serious oversight IMHO, but I digress). This means that you have zero pulse filtering since you're just counting them.

What you should be doing is looping through the JSON construct within the pulse output from your API call to find the key "is_subscribing" and only match pulses that have a value of 1 on this key...this is the way the API 'filters' pulse results for your API key. Notice in my example below the only pulse that matches is a 0, yet I end up with a 1 for the single result.

Also if it has {"source" : "whitelist"} in the validation section, it's not to be flagged, ever. It's there to prevent boneheads from flagging things like Google DNS and OpenDNS from automated sensors and making a huge mess, hence what @GTownson is running into with Google DNS 8.8.8.8 and what I'm running into with OpenDNS 208.67.220.220.

{
  "single_value": 1,
  "multi_value": {
    "sections": [
      ......
    ],
    ......
    "pulse_info": {
      "count": 1,
      "references": [],
      "pulses": [
        {
          ......
          },
          "pulse_source": "api",
           ......
          "is_subscribing": 0,
    ..................................................
    "validation": [
      {
        "source": "whitelist",
        "message": "contained in 208.67.220.220",
        "name": "Whitelisted IP"
      }

Unfortunately none of this can reliably be processed or compensated for in the pipeline constructs due to the lack of loops and control structures. It has to be fixed in the plugin to operate properly. I know hardly anything about Java else I'd submit a pull request (did I mention I'm not a programmer?); hopefully the pointers laid out above will suffice to enable you to make a quick fix.

[dennisoelkers]

Thanks for the valuable input, @jasonkeller. It helps a lot fixing this issue.

[dennisoelkers]

@GTownson, @jasonkeller: I was fiddling with the OTX API all day and to me it seems like the subscription information is not returned properly. At least I was not able to produce a API response where is_subscribing was set to 1 on any pulse that I was actually subscribed to when querying an indicator. Can you verify/disprove this?

[jasonkeller]

@dennisoelkers you aren't going crazy - I'm perfectly able to replicate your results.

Running down the reason why, however, seems to partly be the difference between following an author and following the pulse itself. Most people will subscribe to authors, not individual pulses (like Alienvault themselves). The portion we should be looking at in that circumstance appears to be here...

          ............
          "author": {
            "username": "AlienVault",
            "is_subscribed": 0,
            "avatar_url": "https://otx20-web-media.s3.amazonaws.com/media/avatars/AlienVault/resized/80/unnamed (1).jpg",
            "is_following": 0,
            "id": "2"
          },
          ............

So unfortunately we need to look in both places (following isn't supposed to include it, but subscribing is supposed to pull in all the pulses according to their verbiage https://www.alienvault.com/documentation/otx/subscribing-following-otx-contributors.htm).

But even then, at the moment I'm still seeing a zero in this field even though I know that I'm subscribed to their feed. Looks like this may be a bug that needs to be filed with the Alienvault OTX team.

On the other hand, we could make this more efficient with querying it differently as well...

We could do this first...(to get all the IDs of the subscribed pulses for that user, which is pruned, along with the pulse names and such we may want to enrich the message with later).
/api/v1/pulses/subscribed

Then recurse into the following using the pulse IDs retrieved above...
/api/v1/pulses/{pulse_id}/indicators

And pull out and pre-cache all the indicators we want from the subscribed pulses (you could even add file hashes and the like). Then all the indicators will be pre-cached for fast lookup, and we significantly reduce the amount of API calls that we make to OTX at the same time.

[dennisoelkers]

Thanks for the input and verification, @jasonkeller.

We have a separate caching layer in the lookup table facility, which allows us to cache results quite flexibel, so introducing any caching in the OTX data adapter is misplaced. Tracking the subscribed pulses of the current user to imitate the subscription feature might be an option though, at least as a temporary workaround until the bug is fixed at the OTX side.

I will try to contact someone at their end and see what they are saying.

[jasonkeller]

@dennisoelkers i wasn't clear that i wasn't suggesting caching in the data adapter, merely having the data adapter mass-ingest indicator data from the pulses and preload the cache present in Graylog's lookup table facility.

[dennisoelkers]

I have reached out to AlienVault regarding the API responses and they are looking into it.

[jasonkeller]

@dennisoelkers

I'm not sure if you've gotten a response from Alienvault thus far, however I figured I'd chime in with some additional findings. I'm currently using their Python SDK, and after messing around with it for a while it seems they are pushing people to simply pull in all the pulses they are subscribed to and parse from there. Obviously that wouldn't work too well if you don't have an API key, but when you do have an API key it's looking like the preferred method. I can show you what I'm doing in Python if you like (hopefully it maps closely to their Java SDK).

They also seem to have some modification-date parameters so you can selectively pluck new changes to your pulses after an initial full-sync. Realizing what you're trying to do with Graylog, I think this seems a far better way to do it rather than an individual API lookup per indicator hit; certainly far-more performant.