2019 Technology Predictions

19 For 19: Technology Predictions For 2019 and Beyond

Striim’s 2019 Technology Predictions article was originally published on Forbes.

With 2018 out the door, it’s important to take a look at where we’ve been over these past twelve months before we embrace the possibilities of what’s ahead this year. It has been a 2019 Technology Predictionsfast-moving year in enterprise technology. Modern data management has been a primary objective for most enterprise companies in 2018, evidenced by the dramatic increase in cloud adoption, strategic mergers and acquisitions and the rise of artificial intelligence (AI) and other emerging technologies.

Continuing on from my predictions for 2018, let’s take out the crystal ball and imagine what could be happening technology-wise in 2016.

2019 Technology Predictions for Cloud

• The center of gravity for enterprise data centers will shift faster towards cloud as enterprise companies continue to expand their reliance on the cloud for more critical, high-value workloads, especially for cloud-bursting and analytics applications.

• Technologies that enable real-time data distribution between different cloud and on-premises systems will become increasingly important for almost all cloud use-cases.

• With the acquisition of Red Hat, IBM may not directly challenge the top providers but will play an essential role through the use of Red Hat technologies across these clouds, private clouds and on-premise data centers in increasingly hybrid models.

• Portable applications and serverless computing will accelerate the move to multi-cloud and hybrid models utilizing containers, Kubernetes, cloud and multi-cloud management, with more and more automation provided by a growing number of startups and established players.

• As more open-source technologies mature in the big data and analytics space, they will be turned into scalable managed cloud services, cannibalizing the revenue of commercial companies built to support them.

2019 Technology Predictions for Big Data

• Despite consolidation in the big data space, as evidenced by the Cloudera/Hortonworks merger, enterprise investment in big data infrastructure will wane as more companies move to the cloud for storage and analytics. (Full disclosure: Cloudera is a partner of Striim.)

• As 5G begins to make its way to market, data will be generated at even faster speeds, requiring enterprise companies to seriously consider modernizing their architecture to work natively with streaming data and in-memory processing.

• Lambda and Kappa architectures combining streaming and batch processing and analytics will continue to grow in popularity driven by technologies that can work with both real-time and long-term storage sources and targets. Such mixed-use architectures will be essential in driving machine learning operationalization.

• Data processing components of streaming and batch big data analytics will widely adopt variants of the SQL language to enable self-service processing and analytics by users that best know the data, rather than developers that use APIs.

• As more organizations operate in real time, fast, scalable SQL-based architectures like Snowflake and Apache Kudu will become more popular than traditional big data environments, driven by the need for continual up-to-date information.

2019 Technology Predictions for Machine Learning/Artificial Intelligence

• AI and machine learning will no longer be considered a specialty and will permeate business on a deeper level. By adopting centralized cross-functional AI departments, organizations will be able to produce, share and reuse AI models and solutions to realize rapid return on investment (ROI).

• The biggest benefits of AI will be achieved through integration of machine learning models with other essential new technologies. The convergence of AI with internet of things (IoT), blockchain and cloud investments will provide the greatest synergies with ground-breaking results.

• Data scientists will become part of DevOps in order to achieve rapid machine learning operationalization. Instead of being handed raw data, data scientists will move upstream and work with IT specialists to determine how to source, process and model data. This will enable models to be quickly integrated with real-time data flows, as well as continually evaluating, testing and updating models to ensure efficacy.

2019 Technology Predictions for Security

• The nature of threats will shift from many small actors to larger stronger, possibly state-sponsored adversaries, with industrial rather than consumer data being the target. The sophistication of these attacks will require more comprehensive real-time threat detection integrated with AI to adapt to ever-changing approaches.

• As more organizations move to cloud analytics, security and regulatory requirements will drastically increase the need for in-flight masking, obfuscation and encryption technologies, especially around PII and other sensitive information.

2019 Technology Predictions for IoT

• IoT, especially sensors coupled with location data, will undergo extreme growth, but will not be purchased directly by major enterprises. Instead, device makers and supporting real-time processing technologies will be combined by integrators using edge processing and cloud-based systems to provide complete IoT-based solutions across multiple industries.

• The increased variety of IoT devices, gateways and supporting technologies will lead to standardization efforts around protocols, data collection, formatting, canonical models and security requirements.

2019 Technology Predictions for Blockchain

• The adoption of blockchain-based digital ledger technologies will become more widespread, driven by easy-to-operate and manage cloud offerings in Amazon Web Services (AWS) and Azure. This will provide enterprises a way to rapidly prototype supply chain and digital contract implementations. (Full disclosure: AWS and Azure are partners of Striim.)

• Innovative new secure algorithms, coupled with computing power advances, will speed up the processing time of digital ledger transactions from seconds to milliseconds or microseconds in the next few years, enabling high-velocity streaming applications to work with blockchain.

Whether or not any of these 2019 technology predictions come to pass, we can be sure this year will bring a mix of steady movement towards enterprise modernization, continued investment in cloud, streaming architecture and machine learning, and a smattering of unexpected twists and new innovations that will enable enterprises to think — and act — nimbly.

Any thoughts or feedback on my 2019 technology predictions? Please share on Steve’s LinkedIn page: https://www.linkedin.com/in/stevewilkes/  For more information on Striim’s solutions in the areas Cloud, Big Data, Security and IoT, please visit our Solutions page, or schedule a brief demo with one of our lead technologists.

Striim for Enterprise Security Analysis

Real-Time Enterprise Security Analysis with Streaming Integration

A Holistic Approach to Real-Time Enterprise Security Analysis

By enabling real-time enterprise security analysis, Striim helps companies deliver a comprehensive view of internal and external vulnerabilities, and a more timely and effective response to real threats.

Striim is not meant to replace SIEMs. Striim complements existing enterprise security solutions and traditional SIEM software with an end-to-end streaming integration and analytics platform in support of a holistic security strategy. By ingesting, organizing, correlating, visualizing, and alerting on all security events – automatically, in real time – Striim enables businesses to detect and act on security threats immediately, with fewer false alarms.

In addition, Striim enables users to operationalize artificial intelligence (AI) results via integration with leading AI frameworks, and to validate machine learning models.

Striim for Enterprise Security AnalysisBenefits of Striim for Enterprise Security Analysis

  • Faster response and remediation
  • Fewer false positives
  • Fast, accurate, and detailed delivery of analysis to incident handlers
  • Higher productivity for security analysts
  • Ability to customize alerts and application logic
  • Proactive response to internal and external threats
  • Easier compliance with regulations
  • Support beyond security use cases for real-time operational intelligence

Enterprise Security Analysis Use Cases

Real-Time Security Log Correlation: Striim analyzes multiple endpoint security solutions’ logs and different infrastructure components’ logs in real time to identify issues or exploits that are not obvious from a single security solution. Striim correlates security events (such as a file executing from the recycle bin) with operational logs for context (such as login and logout times and locations). This comprehensive view enables incident assessment in seconds that would otherwise take hours for an analyst to compile and correlate by hand, and helps eliminate false positives.

Alerting On and Continuing Malware Infections: Striim analyzes the logs from the Anti-Virus software and the logs from the IDS that show the compromised computer performing scans and attempting to infect other systems with malware, and issues alarms. By visualizing logs via Striim in real time, incident handles can identify all affected computers at once and focus their efforts on the infected systems immediately.

Real-Time Monitoring and Failed and Invalid Logins: Striim analyzes system logs, and tracks invalid user IDs and invalid passwords by source IP in real time. The platform cross-checks against blacklisted IPs and triggers alarms based on event counts and other custom criteria. Striim can share this data with machine learning software to augment anomaly detection. This rich, live information enables security analysts to quickly and accurately take action such as blocking a problematic IP address.

Core Striim Platform Capabilities 

Striim offers the following key features to support a holistic security strategy:

  • Real-Time Integration for All Security Events: Integrate and analyze from different sources including existing SIEM event logs, network IDS logs, firewall logs, router logs, application logs, as well as sensors, transactional databases; can interface with existing logging systems such as SYSLOG-NG
  • Enterprise-wide, Multi-Log Correlation: Correlate multiple streaming sources to see patterns that point SIEM solutions cannot detect
  • Live Dashboards: Displace rich contextual data on interactive dashboards enabling fast and accurate threat assessment
  • Real-Time Alerts: Send alerts via email, text, and dashboards instantaneously
  • Real-Time Action: Trigger automatic actions to mitigate damage
  • Easy Customization: Allow easy modification to application logic to adapt to new threats
  • Easy Integration: Distribute events and analytics results to downstream analytics, artificial intelligence, and log repositories for broader analytics and compliance purposes.

To learn more about how you can utilize the Striim platform to enhance your enterprise security analysis solutions, visit the Striim’s Enterprise Security solution page, schedule a demo with a Striim expert, or download the platform and try it for yourself.

Using Streaming Analytics to Solve Security Mysteries

Why consider streaming analytics? Because analysts deserve a quiet lunch, too.

“Cause you’re working
Building a mystery
Holding on and holding it in
Yeah you’re working
Building a mystery
And choosing so carefully”

– Sarah McLachlan


Every so often in the life of an SOC analyst, there are moments of perfect clarity. These do not always come from behind a desk, nor in front of a computer screen. Sometimes it’s the perfect afternoon, with the scent of spring in the air; a table in the less crowded section of the lunch room; a perfectly seasoned Italian sandwich with just enough red wine vinegar to know it’s there, but not so much that it soaks the roll and ends up on your shirt. It’s during these moments that people who wear ties tend to notice the serene waters of the afternoon and come rushing in on their Zodiac boats, splashing water all over your moment like a can of LeCroix that saw one too many bumps during your daily commute.

“It seems that there have been some disturbances with one of the servers in the production environment,” The Tie says. Suddenly, and for no clear reason, logs were cropping up with people named “Pi” and “Public,” trying to log in to one of our corporate servers. The Tie asked me most politely to set aside my respite and have a look at what was going on. My love for a good lunch hour was surpassed by a momentary flashback to the movie Hackers when the most common passwords were discussed. With a smile, I told The Tie I would gladly look into this. The Tie smiled that smile from Office Space, and went on his way.

Moments later, in the basement, I sat down and opened a fresh caffeination cylinder and a playlist of Sarah McLachlan songs, and dug in.  The first stop was to look at the auth.log file for the server that had been under attack.

Apr  8 08:19:45 ip-123-45-6-7 sshd[19131]: input_userauth_request: invalid user postgres [preauth]

Apr  8 08:19:45 ip-123-45-6-7 sshd[19131]: Received disconnect from port 41158:11: Normal Shutdown, Thank you for playing [preauth]

Apr  8 08:19:45 ip-123-45-6-7 sshd[19131]: Disconnected from port 41158 [preauth]

Apr  8 08:19:57 ip-123-45-6-7 sshd[19133]: Invalid user test from

Apr  8 08:19:57 ip-123-45-6-7 sshd[19133]: input_userauth_request: invalid user test [preauth]

Apr  8 08:19:57 ip-123-45-6-7 sshd[19133]: Connection closed by port 53513 [preauth]

Apr  8 08:22:37 ip-123-45-6-7 sshd[19135]: Invalid user ftpuser from

Apr  8 08:22:37 ip-123-45-6-7 sshd[19135]: input_userauth_request: invalid user ftpuser [preauth]

Apr  8 08:22:37 ip-123-45-6-7 sshd[19135]: Received disconnect from port 40520:11: Normal Shutdown, Thank you for playing [preauth]

From the looks of this it would seem that someone was trying the old dictionary attack attempting to log in to the server. Many people wonder why these attacks still happen. The simple answer is that they still work. If people did not leave themselves open to attacks like this, Shodan would be nothing more than a rank in the game Go. Still, even with solid gates, multi-factor authentication and security analysts protecting networks, there is no excuse for allowing people to just hammer on the walls. One never knows where they might find a foothold. It was time to start playing the role of analyst.

The best approach to this would be to use streaming analytics, so that we could capture the past and the future attempts in real time for immediate as well as long term analysis. My first step was to create a new Striim application to gather this data from the server, and prepare it for automated processing. With a deep breath, the TQL started to flow.


-- Set workspace  and import JAVA tools needed


IMPORT static NSLookup.*;

-- Create flow for intake of the log file

-- Source : /var/log/auth.log 

CREATE FLOW logflow;

CREATE SOURCE raw_auth_log USING FileReader (





    rolloverstyle: 'Default',

    includesubdirectories: false,

    adapterName: 'FileReader',

    positionbyeof: false


  PARSE USING FreeFormTextParser (

    blockascompleterecord: false,

    parserName: 'FreeFormTextParser',

    charset: 'UTF-8',

    RecordBegin: '\n',

    ignoremultiplerecordbegin: 'true',

    separator: '~',

    handler: 'com.webaction.proc.FreeFormTextParser_1_0'



END FLOW logflow;

Next, I needed to normalize and parse these log entries. To do this effectively would take two steps. First, getting the log into its component parts, and then down to specific events for analysis.

To accomplish the first part:

CREATE FLOW parselogs;

-- Crate type for auth Logs




    _LSTS             DateTime, 

    _LOGSOURCE          String,

    _SERVICEPID         String,

    _MESSAGE            String


-- Parse AUTH log





MATCH(DATA[0],'\\\\n(\\\\w{3}\\\\s+\\\\d{1,2} \\\\d{2}:\\\\d{2}:\\\\d{2})') AS _LOGSERVERTIMESTAMP,






MATCH(DATA[0],':\\\\d{2} (.+?) ') AS _LOGSOURCE,

MATCH(DATA[0],':\\\\d{2} .+? (.+?): ') AS _SERVICEPID,

MATCH(DATA[0],': (.+?)$') AS _MESSAGE


END FLOW parselogs;


This would successfully partition the data and ensure that the next step would be easier to accomplish. Next, we had to determine what we had in our hands. The Tie had mentioned that they had noticed invalid users, so that was a good place to start. Reviewing the log, it was clear how these attempts were captured. Although an invalid user created two log entries, one had more information than the other, and could easily be separated from the rest.

Apr 10 02:35:45 ip-10-99-11-243 sshd[8377]: Invalid user test from

Apr 10 02:35:45 ip-10-99-11-243 sshd[8377]: input_userauth_request: invalid user test [preauth]

Apr 10 02:48:07 ip-10-99-11-243 sshd[8624]: Invalid user sysadmin from

Apr 10 02:48:07 ip-10-99-11-243 sshd[8624]: input_userauth_request: invalid user sysadmin [preauth]

The entries that contained the invalid user id and the source IP address contained the string “Invalid user,” so I could capture those by the capital letter I and not pay attention to the other message with less info. This may seem like a small thing, however when you start working with large amounts of data, everything you can exclude saves you processing time and storage space.

The next step would be to refine the data. The data we are most interested in at the moment is now contained in _MESSAGE. From there we can make a query against our streaming data to extract the invalid user id, and the IP address from which the login request came from. It was time to create a stream that contained this data:


CREATE FLOW analize_logs;

CREATE TYPE InvalidUserType


    _MESSAGE            String,

    _LSTS             DateTime, 

    _USERID             String,

    _IPADDRESS          String







MATCH(_MESSAGE,'Invalid user (.+?) from',1) AS _USERID,

MATCH(_MESSAGE,'Invalid user .+? from (.+?)$',1) AS _IPADDRESS


END FLOW analize_logs;


So with that query in place, I jumped over to my console to ensure that my results were what I was after:





   _MESSAGE = Invalid user play from

   _LSTS = 2000-04-08T06:34:52.000Z

   _USERID = play




   _MESSAGE = Invalid user oracle from

   _LSTS = 2000-04-08T06:58:59.000Z

   _USERID = oracle




So far, so good. The next step would be to gather the data in a meaningful way and enrich it. This is simple, with a few queries. A short pause for knuckle cracking and it was done. A quick dashboard query showed us some useful data about the IP addresses attacking:

And, as expected, there was quite a bit of activity, and some good intelligence to be had:

So what have we here? It seems a large number of IP addresses each trying exactly twice, a few trying exactly 20 times, and two very persistent with nearly 40 tries in our sample. It seems clear that we had attracted the attention of some script kiddies. One thing seemed odd though – why did The Tie just now notice this? Let’s take a look at it from another angle…

A quick review of the offending IP addresses, and enrichment by looking them up in the Maxmind database, it appeared that they were mostly proxies, and from a handful of countries. Once again, the streaming analytics could have saved the day if they had been in place all along. I quickly added these queries and dashboards to the security system, and in doing so I noticed one extra bit of information: the attacks lasted only two days. There was not a trace of them before or after a two-day window. I took this to The Tie, and asked him to compare it against our change control system.

Sure enough, what had happened was that during an update of our AWS environment, someone had made a change to a security group during a ‘testing’ phase, and opened the gates to (anyone). This was spotted and corrected, and noted in the change control documentation. The rest was history. Constant scans against known address space were present and just waiting for an opportunity to open. Thankfully, our well established defense in depth policies (key based logins, change control, logging, removal of default user ID / password combinations, etc… BOFH stuff) prevented any incursion. For good measure, however, I took my newly formed queries and dashboards and updated our log review policy and procedure to ensure that the real time streaming data was going to be analyzed from now on, complete with alarms and access to our paging tree. Streaming analytics had once again saved the day. After finishing my report, I took a deep breath and decided that after a successful day it was time to make my way home, where the most complex system I had to deal with was my cat.

The next morning when I came into my office, I was greeted with a sight that would humble and bring a tear to the eye of any analyst. On my desk was a memo from The Tie, stating what I had found and lecturing the dev ops team about opening up security groups to the world. Underneath the memo was a six pack of Mexican Coke in tall green glass bottles. A post it note attached said “With my compliments,” and it was signed by The Tie.

Questions? Comments? Suggestions for a topic to take on? Send them on to frankc@striim.com and you may find yourself spending an afternoon in The Analyst’s Shack.


How to Utilize Multiple DNS Services to Increase Security

Use Streaming Analytics to Monitor and Secure DNS Services

Once upon a time in a hotel bar in Las Vegas, two security professionals sat down over tall drinks of fine Irish whiskey and one asked the other an age old question:

“If you could do any one thing to improve the security of the internet as a whole, what would it be?”

There was a pause while decades of experience and knowledge silently were reviewed and thought over and then, with a smile, they both replied at the same time:

“Harden DNS.”

At that very moment the piano player at the bar started to play, “Sound out the galleon” and the two security professionals raised their glasses to each other and set off scribbling on bar napkins various ways to harden and better monitor their DNS servers.

This is the story of one such way.

How can you utilize streaming analytics to analyze and alert on DNS traffic within your network? In this blog entry we will cover how to utilize streaming analytics to analyze DNS A record requests and validate them against blacklists and external services for security, and against public DNS services for accuracy. While we acknowledge that each network is unique, this example is fairly generic and should work with most networks where DNS is handled internally. Should your network be substantially different, have heart, because flexibility and the ability to adapt is one of the cornerstones of stream processing. For now, let’s assume we have a medium-sized network in which we will provide DNS service internally, and for our streaming analytics platform we will use Striim.

Step One: Define goals

The goal here is to both improve your security stance and to explore diverse uses of Striim as a security platform. We will do this by analyzing the DNS query logs and comparing DNS A record requests against an internal blacklist, an external blacklist, external domain validation systems, and alternative DNS services. Our goal is to analyze DNS A record requests and to alert the SOC if we discover traffic on the network that indicates communication with known bad domains, questionable domains, non existing domains, or questionable requests.

Step Two : Gather Resources

The resources required for this app are easily found, and you most likely already have this information readily available.

  • Internal Blacklist

Utilizing your various security systems, you have previously identified domains that are known bad actors. These could be domains that belong to VPN services, sources of previous attacks, or just a list of domains from which you don’t care to see traffic from. It can be argued that these can be addressed by blackholing traffic at the border, however, in the spirit of defense in depth, while that can stop traffic from connecting, this app will assist in the effort by providing a secondary check that clearly indicates who is requesting a connection to the undesired domain, and a backup should the primary checking process fail due to hardware/software failure or compromise.

  • External Blacklist

This is a secondary blacklist similar to the internal one, however, this blacklist is curated by an external source. We all have our favorite blogs, projects, and resources. This allows you a place to utilize their results and information while keeping it separate from internal, potentially confidential data on past bad actors against your network. For this example we will be using a externally curated list of domain names associated with ransomware attacks.

  • DNS server logs

For this example we are using SYSLOG-NG to gather logs from our internal DNS servers, breaking them down into individual logs based on content, and storing them in a central logging location. In this case, we are only interested in the DNS Query logs, however, you can use your imagination and the flexibility of Striim to make good use of the other DNS logs provided by your DNS server. Here is an example of a sanitized DNS query log entry:

Aug 22 18:11:04 ns1 queries-log: 22-Aug-2017 18:10:58.461 queries: info: client (press-any-key.foo): query: press-any-key.foo IN A -ED (

  • Java JAR for querying external DNS servers

This will allow us to query external DNS servers for any information that they can bring to our analysis. The requests are made by inserting the following in your TQL:


In this example, we will extract the query from the DNS server query logs into the field _QUERY , and the record request type into _TYPE.

If we were to check the record against the Quad9 database of bad actors, the TQL would look like this:




In this case we are using the query extracted from the DNS query log, querying against the Quad9 database, for only DNS A records.

  • DNS services to compare against:

For this example we will use the following DNS services to compare the results from our internal DNS server against:

We will use these four services to validate the DNS queries that our network generates.

Step Three: Set process and parameters

How we apply the resources we have is as important as the resources themselves. This is where a good flowcharting system comes into play. We also have to create a scoring system in order to determine what, if any, action is to take place based on our results.

For this demo we will keep things simple. We will report either an ‘all clear‘ or no alert, a yellow or caution alert, and a red alert if we have high confidence that bad traffic is happening. There are a number of ways to score this, from this simple 3 color method, to methods involving assigning score values to all possible outcomes. The depth is up to you thanks to Striim being flexible as to how you score and display threats.

As far as process goes, let’s take it one at a time.


The external DNS servers will react differently from each other, so we have to be careful and let our TQL reflect this. Here is a grid showing what you can expect from the servers we are using in this demo:

* Google says that they only block domains under special circumstances. (https://developers.google.com/speed/public-dns/faq)

** Cloudflare offers extremely heightened DNS privacy and DDoS mitigation but not blocking of domains like Norton & Quad9 advertise. For details see (https://blog.cloudflare.com/announcing-1111/)


One of the most important things that we can do with our data is to create a list of bad actors, bad traffic, and unwanted traffic based on our own internal experiences and requirements. From the review and analysis of our traffic, we can create a list of known items to serve as future warnings. Policy and procedure will dictate how often these internal blacklists are reviewed, how they are applied, and to ensure that they perform in an effective way or are corrected so that they accomplish the goal of better security.


There are many groups and organizations out there who are dedicated to identifying and sharing information on bad actors. Many of them provide open source blacklists that can be either directly queried (in the case of Quad9 and other similar) or downloaded, imported, and integrated into a security system.


To effectively use these blacklists, we will load them into Striim by way of a cache, which will be used to compare against real-time traffic in a stream as Striim processes the data. Let’s start by defining the cache and loading our internal blacklist into it:



   _DOMAIN String,

   _IP String,

   _REASON String






   charset: ‘UTF-8’,

   blocksize: ’64’,

   positionbyeof: ‘false’



   columndelimiter: ‘,’,

   header: ‘true’



Next, let’s repeat this process with our external blacklists, utilizing a blacklist for known botnet command and control servers and ransomware servers:


   _DOMAIN String






   charset: ‘UTF-8’,

   blocksize: ’64’,

   positionbyeof: ‘false’



   columndelimiter: ‘,’,

   header: ‘true’





   _DOMAIN String,

   _REASON String,

   _REFATE String,

   _REF String






   charset: ‘UTF-8’,

   blocksize: ’64’,

   positionbyeof: ‘false’



   columndelimiter: ‘,’,

   header: ‘true’


QUERY (keytomap:’_DOMAIN’) OF DNS_C2_Type;



Finally, we can create a dashboard that takes all of this gathered information and places it in a series of comprehensive dashboards so that an analyst can observe the data and make security decisions with clarity and speed.

As you can see, by applying the strengths of stream processing to your log files, you can create tools that not only improve your security stance, but also give your SOC quick access to vital information when during security incidents.

That’s all for now!

Do you have questions about how to use streaming analytics to improve your security posture? Do you have security related data but are unsure how to use or analyze it? Contact me at frankc@striim.com and I will address your questions in future blog entries! Make sure that any information you share is cleared through your security process and procedures before sharing with us.


Customizing Security Solutions for Network Monitoring

Security: Utilizing the Right Tools for Proper Network Monitoring


Everybody’s talking to computers
They’re all dancing to a drum machine
I know I’m living on the outside
Scared of getting caught between”

– Rick Springfield

So far we have discussed the history of the analyst’s craft. It’s time to start cracking open the playbooks and to talk about actual nuts and
bolts of a well-rounded security plan. While there are limitless ways, and no one-size-fits-all solutions, we can take the time to use highly customizable tools and methods to make the best for our needs. The goal has been and always will be fast and effective network monitoring & response to alerts.

The first step in any plan is to assess the network that you will be monitoring and protecting. Through this process we will isolate and identify each part of the network and create a ‘just right’ fit within the security plan for each and every device. Remember that all parts of a network contribute to and reflect the validity of the security posture of the network. 

The major parts of a network can be broken down into three categories:

  • Network Devices

The devices that make up the network upon which all data flows. These are mostly appliances that run their own firmware and have logging options that are configurable by the device administrator. Examples of these devices are routers, switches, and load balancers.

  • Service Provision Devices

These devices are either appliances or servers running specific software that provide services to the local network, the internet itself, or both. Examples of these devices are web (HTTP) servers, Mail (SMTP) servers, and file servers ( FTP … I’m not going to say FTP, and neither should you).

  • Security Devices

These devices are either appliances or servers running specific software that provide security services to the network. Examples of these devices are intrusion detection / prevention systems ( IDS/IPS), firewalls, and antivirus systems.

By breaking the network devices up into these three categories, we provide an analysis space for each to weigh in uniquely in the security plan. For network monitoring, comprehensive analysis and correlation between these three groups many attacks can be detected, followed, and, using the information gathered, proactive measures can be developed on the fly. This also allows the division of responsibilities between multiple analysts so that while on duty they can focus on a single discipline, and by rotation they can be exposed to all three disciplines over time to keep their skills sharp and to distribute knowledge and experience across the team.

One question that gets asked is why we separate the devices between appliances and servers providing services. This is an important distinction because of the way they are designed and implemented on the network.

An appliance is most often a small self contained system that runs on a custom operating system selected by the appliance manufacturer, and is capable of being configured and operated by way of a user interface that is designed by the manufacturer. An administrator will, on a regular basis, review the appliance and keep in communication with the manufacturer to ensure that the firmware and software on the appliance are up-to-date and that any required patches are applied in the proper manner. Logging is accomplished inside of the appliance, with various output methods to central collection points or servers. Because of their closed nature it is very common for the majority of the logs generated by an appliance to be held externally due to the fixed amount of storage space available internally. For the most part appliances need only the configuration for the task they are designed to perform and then to be attached to the network to begin working.

A server running a service is most often a computer, running a selected operating system that runs service providing software. In many cases the selection of operating system and service providing software is done by the company by way of it’s system administrators and other selection committees to best serve the needs of the company. These systems require more administration and closer monitoring than appliances as the responsibility of the care and feeding of the underlying operating system is a designated local responsibility and not left to the manufacturer. An extended version of this is the ever popular virtual server, which consists of one large scale physical computer sectioned off logically into a number of smaller virtual computers, each with its own assigned task. These systems present a unique logging challenge because of the number and depth of the logs generated. The physical server, or host, has the logs generated by its underlying operating system, its virtualization software, and the logs generated by each virtual computer, or guest, that runs within the virtual environment.

That’s a lot of logging to address. Luckily we have time to cover all of this and more. The future is streaming technology, and to be part of that, the craft of the analyst needs to evolve to meet the challenges of big data, cloud processing, and more. Network monitoring will continue be crucial to address these emerging technologies.

In our next blog we will start to drill down into the details of each of the three groups, starting with the network devices, and show how they can at first be overwhelming but can be tamed to provide valuable data.