Cisco Live Melbourne SOC Report

Splunk

Our Splunk stack consisted of Splunk Cloud and Splunk Attack Analyzer. Splunk Cloud had Splunk Enterprise Security (ES) and the Cisco Security Cloud apps installed. Since our security tools include on-premises appliances like the Firewall Management Center and the Secure Network Analytics Manager we needed to be able to get the data from on-premises to the cloud. The solution was to stand up a UCS M3 server that we had on site. Once we got the server online, we deployed a small Ubuntu virtual machine and installed Splunk on it.

The Cisco Security Cloud app, which is published on the Splunk base app store, is a single app to get data from Cisco Security tools into Splunk. The app is modular so individual products can be configured to ingest data into Splunk including Secure Malware Analytics, Firewall, Secure Network Analytics, Cisco XDR and more. The app includes a pre-configured dashboard for each product and health monitoring of the app to see how much data is being ingested. When data is ingested, the app transforms the data to a Common Information Model (CIM) which is Splunk’s universal schema for indexing data. This allows us to create visualizations across multiple data sets or search for a single field across multiple telemetry types.

With the Cisco Security Cloud app configured to ingest data from our various sources we then installed the universal forwarder app to connect to the Splunk cloud deployment. The universal forwarder was extremely performant and was able to forward gigs and gigs of data to Splunk cloud without ever exceeding 30% CPU or a reasonable ingest delay. This allowed us as SOC analysts to search data in Splunk cloud which is also where we had Enterprise Security installed. Incidents from XDR were automatically populated as notables in Splunk ES.

Secure Firewall Threat Defense

The Cisco Secure Firewall (CSF) deployment at Cisco Live Melbourne is an IDS deployment that receives a TAP from the existing network and security infrastructure used by the conference. CSF acts as the traffic ingestion point for the other security tools used by our SOC, collecting valuable data and generating logs and events that are used to inform products like Cisco Splunk and Cisco XDR. CSF also pulled files directly from unencrypted sessions, submitting them to Secure Malware Analytics for sandbox analysis.

Operating in passive IDS mode does have visibility drawbacks, as we lose the ability to use TLS Server Identity to pull additional information from HTTPS connections, and general decryption is off the table. However, the firewall still provides core alerting capabilities, and the dozens of datapoints captured for each connection proved key in many investigations, most notably covered in the ‘Sifting Traffic with Secure Firewall’ and ‘Malware Callouts from the Show Floor’ sections.

From a geolocation perspective, Cisco Live attendees showed a strong prevalence for connections back to the USA, dwarfing all other connection destinations.

The home country of Australia also made a strong showing with twelve million connections. No other country cleared a million connections, but the rest of the list showed an unsurprising prevalence for regional and global tech hotspots. The predictability of geolocation preferences for the attendees allowed us to take a closer look at rarer inbound and outbound geolocation connections, which helped us expand multiple investigations as we looked for additional activity after finding one event. Of course, geolocation data for malicious activity can be faked using Tor, VPN, or a compromised host in another country, but traffic that blends in with expected geolocation patterns is still subjected to signature, heuristic, and sandbox analysis. Geolocation remains one of many characteristics that can reveal attack patterns.

Application data is another area that we monitor at a broad level, in addition to individual alerts for malicious domains. We continue to see plaintext attacks and plaintext information leaks at each conference, but the frequency of these has gradually decreased. At Cisco Live Melbourne 2024, we saw a 15:1 preference for HTTPS over HTTP. HTTP/3 also continues to grow in popularity.

Also of note is the use of DNS over HTTPS to mask DNS requests. While the great majority of DNS requests continue to be plain text, the use of DNS over HTTPS continues to rise. Eventually, we expect to see plain text DNS requests overshadowed by encrypted DNS protocols, much like HTTP is eclipsed by HTTPS today.

Automations

By Aditya Raghavan

On the automation front, we introduced three new automation workflows to help speed up threat hunting for our analysts. Credit to Ivan Berlinson, our colleague from France, for the first two workflows in XDR automation with Secure Malware Analytics, and Adi Sankar for the workflow with Umbrella.

1. Malicious samples submitted in Secure Malware Analytics

We want to reduce the number of dashboards pivots our analysts deal with. So, for any samples submitted to Secure Malware Analytics that are convicted as malicious (threat score > 90) and seen in the Cisco Live environment, this automation workflow would automatically create an incident in XDR and send a Webex message to the Incidents channel. The above is an example. While this isn’t something to do in a production environment every time, it is useful for bubbling up interesting avenues of investigations right in XDR and Webex to our analysts.

2. Non-malicious samples from common document formats

Similarly, we typically see some content transmitted in clear text at such events. Any documents with common file types submitted to Secure Malware Analytics having a non-malicious verdict (threat score < 85), seen in the Cisco Live environment and of the following types typically have content in clear text. This is worth an investigation for our analysts to identify if there was any critical information being leaked inadvertently. This workflow would automatically create an incident in XDR and send a Webex message to the Incidents channel for documents of the following file types.

• PDF, TXT, XLS, XLSX, XLSM, PPT, PPTX, PPTM, DOC, DOCX, DOCM 

3. Create incidents from Umbrella Security Events

Any DNS Security Events in Umbrella for certain categories of interest would be brought forward to the analyst as an incident per category. This shows an example of an automation created incident for the Malware category.

Analyst Stories

CoinLoader Infection Investigation

Christian Clasen

A couple days into the conference we noticed several block events in Umbrella DNS. The events were TXT record queries for what appeared to be randomly generated subdomains belonging to ucmetrixsdn[.]info. The queries resemble the domain generation algorithm (DGA) technique commonly deployed for malware beaconing.

DGA is a technique in command and control (C&C) infrastructure that generally serves one of two purposes: to retrieve instructions from the malware’s authors or administrators, or to exfiltrate data from the infected endpoint through covert channels. Because this malware is well-known (first detected in 2018), we can use public intelligence to compile expected behaviors and additional indicators of compromise to begin our investigation.

The DGA behavior here is well-known and attributed to the CoinLoader malware. Dark Trace has a detailed write-up that provided us some direction: https://darktrace.com/blog/catching-coinloader-decrypting-the-malware-hijacking-networks-for-cryptomining-operations. The questions we were immediately looking to answer were:

1. What was the current stage of the attack?
2. Was there any risk to other attendees?
3. Had the user been infected while on the conference network?
4. Who was the user of the infected machine?
5. Were there other related infections at the conference?

CoinLoader is an initial dropper designed to pull down other malicious payloads including ransomware, information stealers, and cryptominers. It seemed that this particular infection was likely at its initial stage, and Umbrella was successfully preventing further stages of infection by blocking the C&C traffic. There was no traffic logged between this device and other attendee IP addresses, nor any scanning activity so the risk to other attendees was presumed to be low.

The CoinLoader malware finds its victims by masquerading as cracked or pirated versions of legitimate software. To determine if the malware was downloaded on the conference network, we searched our SOC tools (including Secure Malware Analytics and Firewall file events) for instances of the file extensions RAR and ZIP, and any instances of filenames containing the strings “keygen” or “crack.” We found no evidence that the malware was downloaded while on the conference network. Because we do not decrypt attendee traffic, this is impossible to know for sure.

To find and notify the owner of the device, we used standard fingerprinting techniques. DHCP logs and traffic patterns are valuable for determining the OS and device type. In this case, MDNS queries emanating from the device gave away both the operating system type and the hostname. The hostname contained the first name of the attendee. Using data from the wireless infrastructure, we were able to physically locate the device on the show floor.

With the user notified and the device triaged, we turned to further hunting of related IOCs elsewhere on the network. We had a few things to look for including:

• A specific string in the Issuer field of the TLS certificate
• A specific ASN and publicly routable IP range located in Eastern Europe.
• Addition C&C domain names and URLs.

Using Splunk, we were able to efficiently search all our log sources for these IOCs and found no other instances of this infection.

Strategies for Client Attribution on Public Wi-Fi

Christian Clasen

Real world deployments often fall short of the idealistic architectures intended by vendors. Events, budgetary and time constraints, and technical feasibility often conspire to prevent the maximalist approach to security infrastructure. When inevitably faced with these challenges, analysts must rely on correlation techniques to make the most of the information available in the SOC environment. One such limitation we faced in the Cisco Live SOC was the lack of Umbrella Virtual Appliance (VA) integration leading to a blind spot in our client-side IP visibility. With a bit of knowledge of the mechanics of Umbrella operation, analysts were able to attribute malicious or suspicious DNS queries to client IP addresses on the public Wi-Fi despite the lack of VAs.

Umbrella is a recursive DNS resolver that utilizes the power of the global DNS to enforce security and acceptable use activity. The public IP addresses in use by the conference are registered to an Umbrella organization so that DNS queries can be attributed and handled by the right policies. Because of NAT, any IPv4 queries will be attributed to the public address servicing all attendees. In an optimal Umbrella deployment, internal recursive resolver would be installed (VAs) and these would provide internal IPv4 attribution. Unfortunately, the internal resolvers used at the conference did not provide this functionality, and so Umbrella alerts only provided public IP address attribution.

The obvious solution to this would be to ingest the internal recursive resolver logs into our SIEM and SOAR infrastructure. This was planned and being actively worked on, but not immediately available in the earliest parts of the conference. So how to bridge this gap and ensure the most specific information is available for these events? The answer is simple if you know how Umbrella works.

When Umbrella determines that a query is for a malicious domain, it doesn’t simply refuse the resolution or return an NXDOMAIN response. It instead resolves to dedicated IP addresses owned by Cisco, and then waits for the subsequent connection so that it can return a block page. For HTTP/S connections, this is the best way to communicate to the end user why their connection failed. Umbrella reserves specific IP addresses for domain categories such as Malware, Phishing, and Command and Control traffic: https://docs.umbrella.com/deployment-umbrella/docs/block-page-ip-addresses.

Armed with this information, there are two strategies for correlating the Umbrella DNS events with Firewall events. By filtering the Firewall connections for the destination IP address associated with Umbrella Malware blocks (146.112.61[.]107) we can find any connections the client subsequently made after resolving the malicious domain. If the connection is attempted over HTTP or HTTPS, we can very likely see the hostname in the HOST header or Server Name Indication (SNI) extension field. This is because the client still thinks it is connecting to the intended malware server, and not Umbrella.

For non-web traffic we can simply correlate the timestamp in the Umbrella event with the IP connection in the firewall events to determine with confidence that the specific internal client IP was the source of the malicious or suspicious DNS query. From there, geolocation information from the wireless infrastructure can help us track down devices and individuals when the content of the alert warrants it.

Scraping Infra Servers

Aditya Raghavan, Adam Kilgore

It all started with Adam seeing a bunch of SSH connections from an IP in the DC static host group range to some internal IPs on a non-standard port (TCP 830). Prima facie, all those connections were successful, so it appeared legitimate.

We investigated the source and destination entities in XDR Investigate and it found another neighboring device from the Infra Management host group also involved in similar traffic patterns. Additionally, the traffic between the devices in Infra Management and DC Static host groups triggered a bunch of Snort signatures on the firewall.

Secure Network Analytics validated the traffic patterns with Fake Application Detected events. This was then escalated to the NOC team as the Infra Management segment was under their ownership.

Freddy Bello, the NOC lead, investigated it and identified the entities as Wireless LAN controller (in Infra Management) and DNA Spaces Controllers (in DC Static). And the traffic pattern involving SSH on a non-standard port was an app on the controller poking them to extract telemetry regarding the status of the access points on the show floor.

While the traffic turned out to be expected, this is a good example of SOC workflows to investigate traffic patterns that appear abnormal or could be a sign of compromise or malicious activity if they are not confirmed to be from a legitimate source. By keeping a close working relationship with the NOC, we are able to provide insights into traffic patterns and behaviors and receive back confirmation of whether an investigation should be escalated or whether it can be safely closed. All in all, this turned out to be a Cisco Live Positive. On to find the next needle in the haystack, folks.

Suspect Data Loss and Port Abuse Incident

Zoltan Karczag, Cam Dunn, Christian Clasen

The SOC received notification from the NOC of some activity that was seen by them on their WAN router:

This activity was dropped by an ACL on the WAN router and never made it to the firewall, so was not seen by the SOC.

A reverse lookup of the IP address identified that the traffic was as originating from Russia:

As a consequence of the above, the onsite NOC’s own investigation into this resulted in an XDR incident seen on 12/11/2024, with the title as per the story title. See screenshot below:

Investigation of the incident confirmed that the NOC initiated a port scan from an internal IP address to the WAN link.

Another Cisco Live Positive.

Suspicious User Agent on

Christian Clasen, Zoltan Karczag, Cam Dunn, Ricky Mok

Multiple incidents seen in XDR of suspicious user agent for various IP addresses in the Cisco Live event internal IP address range.

Investigation shows that It’s due to an (likely Android) application with a poor implementation of the OkHTTP client library (https://square.github.io/okhttp/). The developers of the app are not properly setting or calling the “project.version” variable in their app.

It is most likely to be something running on this e-commerce platform https://open.lazada.com/

The server side implements Octopus https://octopus.com/docs/octopus-rest-api

Investigation via Secure Malware Analytics shows the following:

Via XDR Investigate: 

We lowered the priority in Network Analytics on the suspicious user agent to reduce the number of alerts in XDR for the valid benign user agents detected.

Further refinement could be completed by blocking/filtering the specific observed user agent.

Suspected Phishing Domain

Adam Kilgore, Zoltan Karczag, Tony Iacobelli

• Cisco XDR Alerted on a possible phishing domain that was observed by a host on the network

The SOC used Splunk Attack Analyzer to interact and analyze the website in a safe way, analysis returned a “404 page not found” site when the URL was triaged.

Through further investigation we were able to validate that the top-level domain and associated public IP were owned by “knowbe4” which is a security company specializing in phishing simulation and training.

According to this we identified potential Cisco Live attendees that had just failed their organization’s phishing assessment.

Sifting Traffic with Secure Firewall

Adam Kilgore

A lot of modern analytics work is driven by automation, and rightly so—the Melbourne SOC benefited greatly from the advanced correlation provided by the Cisco Splunk and Cisco XDR platforms. The tremendous amount of data observed and collected by Cisco Secure Firewall is instrumental in feeding those advanced analytics platforms. In addition, the data is also valuable in its own right, and I’m a personal believer in checking datasets for the unexpected.

We can check for unexpected traffic by testing assumptions. One assumption we could make is that port 443 traffic will be HTTPS. Secure Firewall offers the logging, application detection, and search granularity to verify, using a search like the one below:

If the query returns nothing, then we proved our hypothesis—all the 443 traffic in our logs is HTTPS. But if the query does return logs, then we might have something worth looking into, and at the very least something we’ll want to understand. For Melbourne Cisco Live, our search did return some logs:

We can see from the above that we have some HTTP traffic running over port 443. That’s not expected, so it’s worth digging into it a little more to see if we can figure out why it’s occurring and whether there is any security concern. Since the traffic is HTTP protocol, we can check the URL field in the logs.

The URLs above specify a destination IP and port 443, but some also append a path. Of particular note is “./env.” If improperly configured, the “./env” path on a server can reveal sensitive information that could lead to the compromise of the server and serve as an entry point towards a more serious attack. By narrowing down a large subset of expected traffic (HTTPS over port 443) we’ve isolated a much smaller subset of unexpected traffic (HTTP over port 443) that also has a high concentration of malicious activity.

There are two things we can do with this data: (1) look for other malicious activity from the same actors, and (2) confirm whether the “./env” requests successfully retrieved sensitive information from the servers. For (1), an easy method is looking for other activity from the same IP addresses, but this is limited since an attacker can alter their IP address using Tor, a VPN, or a compromised host that acts as a jump server from which to launch attacks. However, even if the attacker varies their IP address, sometimes we can still tie an attack to an individual actor by collecting a unique or semi-unique identifier from a known attack (like a user agent) and then checking for the same identifier in traffic from other IPs. For (2), we can easily determine whether the attack was successful by looking at the packets in the server response, but these won’t be available unless we were running a packet capture when the attack transpired, or if we have a data lake that captured the connection.

If we don’t have the luxury of a packet capture, we may still be able to determine whether the attack was successful using the firewall logs. If we expand our firewall log search to include the packets and bytes columns, we can determine even more about the attack and what data was returned.

Using the packet fields, we can see that most of the connections have seven Initiator Packets. For HTTP, the packets from the initiator IP will be a SYN for the first packet, a SYN/ACK for the second packet, and then a GET request in the third packet. This third packet is the URL we see in the logs above, trying to retrieve the “./env” data in some of the requests. Similarly, in the Responder Packets column, we can expect an ACK for the first packet, and then a response to the GET request that returns some kind of information in the second packet. Our concern is that the information returned for the “./env” requests is different than the data returned from the non-malicious GET request to the server, and whether that response contains sensitive information. Can we determine whether this is occurring just based on the logs? We can, by looking at the bytes. For all the requests above, we see the response is five packets, and the Responder Bytes are always 346 bytes. This tells us that the server is returning the same response to each of the requests, or something very close, for each of the requests in the logs, some of which are trying to access “./env” and some which are not. If the server did return server data for the “./env” request, we would expect to see a variation in the Responder Bytes.

Unsecured Transmissions

Jessica Oppenheimer

At each event, it is common to observe documents containing business records, financial data, or personal identifying information. When possible, we locate the persons affected by the inadvertent disclosure over the network and help them secure their communications. Often it is an insecure email protocol or open network connection, such as http over port 80 instead of https.

A conference is a great place for networking, securely. We saw a CV was accessed and detonated in Secure Malware Analytics. Investigation found the server was not transmitting the data over an encrypted connection.

In another case, business records were transmitted in the clear, again from a web connection over http.

Look for the secure connection icon in your browser and check your email settings to ensure POP3 or IMAP are not mistakenly selected.

We also used the Glovebox feature in Secure Malware Analytics to investigate websites that conference delegates attempted connection, such as this seized domain by law enforcement.

We were able to explore the behavior of websites (such as dropping malicious JavaScript files) without our analysts becoming infected.

Also, the analysts can review the Runtime Video to understand the user experience.

Umbrella DNS request in category Malware

Adam Kilgore, Zoltan Karczag, Ricky Mok

XDR automation via Umbrella connection Identified number of malicious domains connected to by an internal host on the IPv6 network since Nov 11^th, 2024. the observed behavior continues active on Nov 12^th, 2024

Evidence captures on XDR that list the malicious domains and hash values.

Suspicious Callouts from the Show Floor

Adam Kilgore and Christian Clasen

We picked up some DNS requests to a domain previously associated with an Iranian APT and multiple strains of malware.

A DNS request is just one IoC in an investigation. With a full enterprise deployment, we would want to track down what application made the request, when it was installed, and whether there was a legitimate example of user activity that could explain the DNS request and confirm it as not malware related. Since we don’t have endpoint security resources at our disposal for guest wireless connections, and given the potential severity, we decided to see whether we could identify the end user device and notify them of the potential compromise.

However, our lack of endpoint control makes identification difficult as well. The guest wireless connection is provided for free, without requiring individual login credentials or MFA. Where we could normally fall back on authentication logs from services like Active Directory and ISE, at the Melbourne SOC we had to tie the IP back to an identity going purely off network activity. Is that possible? In this case, it was possible using logs from Secure Firewall.

We put a lot of trust in the security of applications and cloud services. While the encryption of these services is usually well configured, they can still share quite a bit of identifying information in those same encrypted sessions. In the above example, both an organization app and an organization SharePoint instance revealed a specific vendor. And while we didn’t see it here, other applications like Slack will also reveal the room that a user is connecting to in an encrypted session. Is that a problem? Yes and no. The contents of the connections are encrypted and secured, but someone with traffic sniffing capabilities (like we have via our TAP in the SOC) can still use that secure connection to tie traffic back to an organization, an individual, or an executive role. A malicious actor could then target the identified organization, group, or executive via their now known IP. Or in our case, we can use the datapoints of the potential malware callout, the company app, and the company SharePoint to notify someone that their device could be compromised.

So, we now have an IP and a vendor name. Time to hit the show floor. We found the booth of the vendor and asked them to confirm whether one of their devices had the IP that made the DNS request—an ipconfig confirmed they did, which was not surprising given the connections made to the SharePoint and company app. We notified them of the DNS requests that started the investigation and recommended that they treat the device and the associated accounts as potentially compromised.

Special Thanks

Acknowledgments 
Thank you to the Cisco/Splunk SOC team:

Senior Analysts
Christian Clasen, Justin Murphy, Aditya Raghavan, Adam Kilgore, Tony Iacobelli, Jessica Oppenheimer

Intern Analysts
Cam Dunn, Milin Mistry, Ricky Mok, Zoltan Karczag, Alex Chan

SOC Leads
Shaun Coulter, Aditya Sankar, Ryan MacLennan

NOC Leads
Freddy Bello, Andy Phillips, Darren Nirens 

Cisco Marketing
Vanessa Carlson!! Lauren Frederick, Trish Stallone

Also, to our SOC partners for licensing

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Security Social Channels

Instagram
Facebook
Twitter
LinkedIn

Share: