Why Go is awesome for building security clients
After reading the great blog post about GO for DFIR by Scott J Roberts , I wanted to elaborate more on using Golang for integrating with various Security products based on my experience at Demisto .
We (Demisto) are building the next DFIR platform, by focusing on automation and collaboration aspects of it, as such we needed to create a platform that will serve as a fabric between many security products (Splunk, Mcafee ESM, FireEye, VirusTotal just to name a few), and since we are using GO as our main code language for that platform, we created many GO clients to interact with those products.
From my experience, GO is really powerful language to create such integrations, it’s has many great built in libraries (like http, tls) and cool features like defer statement , go routine and multiple return values, which makes the code more concise, and faster to write.
And here I will go through a concrete example, ThreatExchange client I contributed to Facebook few months ago (full code here).
Few words on ThreatExchange, it’s an open source project by Facebook, that provide companies a way to share threat intelligence .
We will start with a struct that holds the fields our client needs, in our case appId+appSecret to authenticate against ThreatExchange service, logger to support external app logging, and of course http client to make out REST requests .
Then we add a function to initialize the the struct :
In ThreatExchange API, I have different calls I want to support, like querying Malware Families or Threat Indicators, all use GET http request, so I better create a generic function to support all this calls, and avoid code duplication.
Let’s go over the params here:
1. apiVersion : The api has versions , so for supporting different api version calls I need to make it configurable.
2. resource : this is the REST api resource I should use in my URL
3. startTime + endTime : must calls have time range params optional
4. resourceType + text : data to query by .
5. limit: limiting results
6. extraParams : so i f we have a functions that has unique params.
7. result : this is the result object, this should be a pointer to an empty struct , to be filled by the response body.
Let’s deep dive into this this func implementation (please note I will ignore some error handling here, to make post more compact).
Building the url :
Adding the token to access the API:
I’m using here Sprintf utility function , to format the authentication token.
Adding params if needed (will just mention the endTime param for this example) :
Encoding and building path :
Here is where the actual http call get’s invoked:
Important, don’t forget to close the body to avoid leaks, notice i’m using here defer keyword, this will ensure this line of code will get called in the end of current function call, even if some exception terminates it:
Check for wrong return code in response , if so print it to utility error function:
And finally reading the Json body, and unmarshmaling it to the result pointer:
so now that we have a generic query func, let’s create one function that uses it, GetThreatIndicators :
Notice that i’m using ThreatDescriptorResults struct, that as special struct I created to represent the Json result expected from the API call.
As for the return results of the function, the first item will be the actual struct, and the second will the raw JSON as string.
The ThreatDescriptorResults struct will look like this:
Notice the Json tag, it’s here to describe how to transform the Json keys to the struct, the Data field will look at the Json key ‘data’ and transform it to type array of ThreatDescriptor (which is a complex type).
The Next field is a simple string field, and since it’s not mandatory, I can add the “omitempty” tag, that will drop this field from Json representation if non existent.
AS you can see, built in support in Json in the language is really powerful , and saves you to write much boiler plate code of Json transformers.
And last, let’s give an actual example of using this new library :
This is it :)
All we are left is adding a tests for this code, you can find test example here , but I will elaborate on that more on my next post.
This example is a simple one, only using GET REST request, in next posts I will try to deep dive on more complex clients , which we are looking to open source at Demisto , so keep track of our Github repo at : https://github.com/demisto .
