Short guide for installing Elastiflow on CentOS 7. There are 4 main components - Elasticsearch, Kibana, Logstash, and Elastiflow itself. This guide is quite old now and I wouldn't recommend following it.

NOTE: If you run into any issues, checkout the official elasticsearch documentation! Things can and do change: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html. Highly recommend watching Rob's (Elastiflow creator) YouTube video which I have made full use of in the creation of this article https://www.youtube.com/watch?v=gZb7HpVOges&t=483s.

Installing ElasticSearch

To begin, you'll need to download and add the PGP signing key to your system:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Following that, you'll need to create a file pointing to the RPM repository:

sudo nano /etc/yum.repos.d/elasticsearch.repo

Fill the file with the following data:

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

Once saved, run the following two commands:

sudo yum clean all
sudo yum makecache

Finally, install elasticsearch which is one of the prerequisites for Elastiflow:

sudo yum install --enablerepo=elasticsearch elasticsearch

You can verify the installation with the following:

rpm -qi elasticsearch

Name        : elasticsearch
Epoch       : 0
Version     : 7.9.3
Release     : 1
Architecture: x86_64
Install Date: Mon 09 Nov 2020 20:03:53 GMT
Group       : Application/Internet
Size        : 511650720
License     : Elastic License
Signature   : RSA/SHA512, Fri 16 Oct 2020 13:46:36 BST, Key ID d27d666cd88e42b4
Source RPM  : elasticsearch-7.9.3-1-src.rpm
Build Date  : Fri 16 Oct 2020 11:45:04 BST
Build Host  : packer-virtualbox-iso-1600176624
Relocations : /usr 
Packager    : Elasticsearch
Vendor      : Elasticsearch
URL         : https://www.elastic.co/
Summary     : Distributed RESTful search engine built for the cloud

Now you have a very basic Elasticsearch install, but there's a few default settings that need to be adjusted. The first, is the elasticsearch JVM heap size. The recommended by Rob, the creator of Elastiflow, is roughly 1/3rd of your total system memory. I'm running a bit of a weird setup with 40GB of RAM, so I've assigned 12GB to the JVM heap.

In order to do so, enter the Elasticsearch (ES), JVM options file:

sudo nano /etc/elasticsearch/jvm.options

Where the below two parameters are, adjust to match the size that fits your system:

-Xms1g
-Xmx1g

TO:

-Xms12g
-Xmx12g

Next we need to increase some of the default system limits for our installation. (Official documentation: https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html)

For CentOS 7, we must configure these inside systemd. Firstly create the below directory:

sudo mkdir /etc/systemd/system/elasticsearch.service.d

And now create the overrides configuration file:

sudo nano /etc/systemd/system/elasticsearch.service.d/override.conf

Fill this file with the following system overrides:

[Service]
LimitNOFILE=131072
LimitNOPROC=8192
LimitMEMLOCK=infinity
LimitFSIZE=infinity
LimitAS=infinity

Rob goes into good detail about all 5 of these settings in his YouTube video linked above.

Note that the next step used to be to add the following... to remove the default operating system limits on mmap counts. But this is done by default now as part of the RPM or deb package installation (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html). You can verify the value is the orrect 262144 by running the below:

sysctl vm.max_map_count

Output should be:
vm.max_map_count = 262144

We can now edit the Elasticsearch configuration itself. Add the below lines which will enable basic authentication features, and again increase some of the default search parameters to ensure that ES scales to Elastiflow's needs. the rest can be configured how you like it. For most users you'll be fine with the defaults.

Enter the config file:

sudo nano /etc/elasticsearch/elasticsearch.yml

Add the below:

indices.query.bool.max_clause_count: 8192
search.max_buckets: 250000
action.destructive_requires_name: 'true'

xpack.monitoring.enabled: 'true'
xpack.monitoring.collection.enabled: 'true'
xpack.monitoring.collection.interval: 30s

xpack.security.enabled: 'true'
xpack.security.audit.enabled: 'false'

node.ml: 'false'
xpack.ml.enabled: 'false'

xpack.ilm.enabled: 'true'
xpack.sql.enabled: 'true'

xpack.security.transport.ssl.enabled: 'true'
discovery.type: single-node

Now complete a daemon-reload:

sudo systemctl daemon-reload

Ensure that Elasticsearch starts on boot with the below, and then start ES:

sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Confirm ES is operational:

sudo systemctl status elasticsearch

######### sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─override.conf
   Active: active (running) since Mon 2020-11-09 22:08:34 GMT; 36s ago
     Docs: https://www.elastic.co
 Main PID: 10527 (java)
    Tasks: 106
   Memory: 13.0G
   CGroup: /system.slice/elasticsearch.service
           ├─10527 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -...
           └─10768 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Next we're going to want to setup some passwords for the default accounts in Elasticsearch. Run the below:

sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

At this stage it will ask you to set a password for a number of user accounts - make sure you jot these down. Preferably in your cloud based credential locker of choice 😁.

Finally, you can login to Elasticsearch through your browser using the "elastic" account (that you just created a password for) to verify everything is perfectly operational:

Installing Kibana

At this point you can install the visualization component of the elastic stack, Kibana. Installation instructions differ from earlier documentation - so if you run into any issues, again check the official docs (https://www.elastic.co/guide/en/kibana/7.9/rpm.html)

Firstly create an RPM repo file for Kibana:

sudo nano /etc/yum.repos.d/kibana.repo

Fill with the following information:

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Save the file, and now you can run the package installer:

sudo yum install kibana

Following installation, we can now edit the configuration file for Kibana:

nano /etc/kibana/kibana.yml

By default, nothing is explicitly configured. Below are the explicit lines of configuration that I've added to my setup:

server.name: 'toob-netflow-kibana'
server.host: '192.168.130.20'
server.port: '5601'

server.maxPayloadBytes: 8388608

elasticsearch.hosts: ['http://192.168.130.20:9200']
elasticsearch.username: 'kibana_system'
elasticsearch.password: 'XzeEQ95kpSPaqYD6'

elasticsearch.requestTimeout: 132000
elasticsearch.shardTimeout: 120000

kibana.index: '.kibana'

xpack.security.enabled: 'true'
xpack.security.audit.enabled: 'false'
xpack.monitoring.enabled: 'true'
xpack.monitoring.kibana.collection.enabled: 'true'
xpack.monitoring.kibana.collection.interval: 30000
xpack.monitoring.ui.enabled: 'true'
xpack.monitoring.min_interval_seconds: 30

xpack.apm.enabled: 'true'
xpack.apm.ui.enabled: 'true'

xpack.grokdebugger.enabled: 'true'
xpack.searchprofiler.enabled: 'true'

xpack.graph.enabled: 'false'
xpack.infra.enabled: 'true'

xpack.ml.enabled: 'false'
xpack.reporting.enabled: 'false'

xpack.spaces.enabled: 'true'
xpack.spaces.maxSpaces: 1000

OK now, I ran into an issue whereby my Kibana configuration didn't want to start up. It needed to be pointed to this configuration file we just created. Do so by updating the below file:

sudo nano /etc/systemd/system/kibana.service

Update the following line to match the below:

ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"

Now enable Kibana to run on startup & start Kibana itself:

sudo systemctl enable kibana 
sudo systemctl start kibana

Now you should be able to browse to your Kibana installation on the default port 5601. Login with the default user elastic:

Alright now you're in... it's time to setup the next component of the elastic stack.

Installing Logstash

Before getting to the actual Logstash setup, you'll need to make sure that Java is installed or Logstash installation will fail:

sudo yum install java-1.8.0-openjdk

Verify that Java is installed successfully:

java -version

## Output ##
openjdk version "1.8.0_262"
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

I had to set the JAVA_HOME variable now. First you need to find where your JDK installation is:

sudo alternatives --config java

Copy the full path (/usr/lib/...bin/java), and now create a java.sh file under profile.d:

sudo nano /etc/profile.d/java.sh

Enter the below:

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java

Save, and reboot your system. Once rebooted, run the below to check JAVA_HOME is set correctly:

echo $JAVA_HOME 

This command should output the directory you just put into your java.sh file.

Moving on to Logstash

You should be fairly familiar with the installation process now, this component is no different than the other two 😊. Start by creating a repo file so you can access the logstash repository:

sudo nano /etc/yum.repos.d/logstash.repo

Enter the below into the file:

[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Save, and now you will be able to install Logstash:

sudo yum install logstash

Now that it's all installed, we're going to configure the JVM options for Logstash to increase the heap size allocation:

sudo nano /etc/logstash/jvm.options 

I've given 8GB in my config by adjusting the following (remember my system has 40GB of RAM - heap size for elasticsearch should be roughly 1/3rd your system memory, I'm setting logstash to be 2/3rd's of that number - note that, this should never be set above 31GB):

-Xms1g
-Xmx1g

TO:

-Xms8g
-Xmx8g

Now we should increase Logstash system priority by editing the below:

sudo nano /etc/systemd/system/logstash.service

Adjust the "nice" parameter to be 0, which will increase Logstash process priority to the default.

Now, we need to install and update the following plugins in order to use Elastiflow:

sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp
sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-tcp
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-geoip
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-translate

Some of these might not actually need to be updated, but still it's worth doing.

Installing Elastiflow

You've finally made it to the point where Elastiflow can be installed. There's still settings to tweak, but it's time to download the git repository.

cd #Home
mkdir flowtemp
cd flowtemp
sudo git clone https://github.com/robcowart/elastiflow.git
sudo cp -arv elastiflow/logstash/elastiflow/. /etc/logstash/elastiflow/
sudo cp -arv elastiflow/logstash.service.d/. /etc/systemd/system/logstash.service.d/

Now configure the below file:

sudo nano /etc/logstash/pipelines.yml

And add the following two lines:

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

Now you're good to go.... There are a few other optional steps you can check if you want to customize a bit further. I highly recommend you read the install page for Elastiflow: https://github.com/robcowart/elastiflow/blob/master/INSTALL.md. Note that Geo IP databases in Elastiflow haven't been updated since 2019 - due to new laws in California apparently. Highly recommend creating a MaxMind account and downloading the respective GeoIP database. Again this is covered in the install page above.

Run the below to ensure any changes to the environment variables are recognized:

systemctl daemon-reload

Now start Logstash:

systemctl start logstash

The next step is to download the Kibana dashboards. You can download this file here: https://github.com/robcowart/elastiflow/blob/master/kibana/elastiflow.kibana.7.8.x.ndjson. Note that, this version will change as Elastiflow is updated. Check the github repo to get the latest copy (under "kibana/")

Log on the server web page and navigate to Management/Saved Objects. Import the above file.

With any luck - everything should now be working. You can go into Kibana and select a dashboard to see your incoming netflow data (assuming you have already set this up and pointed it to the server).

Over the last couple weeks I've been attempting to setup and test DHCPv6 Active Leasequery between two relay agents (Juniper MX10003 routers running 19.4R1-S2.2). ALQ is a fairly new extension to DHCP (RFC7653 (v6) and RFC7724 (v4)) which allows for a requestor to ask for information of another devices DHCPv6 bindings, be that another DHCP server or relay agent. In my case, I've got two JunOS DHCP relay agents which are holding DHCPv6 bindings - they need to hold the exact same routing and lease information in order to provide redundancy to the DHCPv6 client. ALQ assists by ensuring the leases remain synchronized between the two routers.

Before going further, see below the design I'm using for testing (omitted some of the lab access components as they're not relevant to DHCP traffic flows).

DHCP Client to Server Traffic Flow

Before going further, it's necessary to understand the traffic flow from the CPE, to the relay agents, to the servers, and the exchange of messages that takes place! So lets look into that in detail. Here is a diagram of the basic traffic flow with a single MX, I'll explain the flow in further detail with both of the Juniper routers below.

When the CPE boots, it will broadcast a DHCPv6 Solicit packet that will be received and processed by both R1 & R2 relay agents. Both agents will forward the solicits to both DHCP servers, in a Relay-Forward message, encapsulating and forwarding on the clients initial solicit message. Here I have two KEA DHCP servers set up with a redundant configuration, therefore only the elected primary server will respond to the Relay-Forwards with a Relay-Reply which advertises to the client the all important DHCP IA_NA and IA_PD prefixes (Info).

So, these Relay-Reply (Advertise/Offer) messages are received by both R1 & R2 from the primary DHCP server, and then both are forwarded to the DHCP client. The client will in-turn send a Request message back towards the primary DHCP server, which is accepting the lease advertisement/offer. Both relay agents will receive the client's request message and forward to the primary DHCP server in Relay-Forward packets. The server will respond to both request messages, finalizing the lease between the client and server, sending two final Relay-Replys towards R1 & R2.

It is at this stage, that R1 & R2 will install the lease into their JunOS subscriber database, and also install routes in their routing tables for the IA_NA and IA_PD prefixes - enabling connectivity to the client CPE! The final Relay-Reply is also forwarded to the client by R1 & R2, and at this stage the client will install it's new IPv6 prefixes.

In summary:

   •  CPE boots - once ready it sends a DHCPv6 Solicit message

   •  R1 & R2 encapsulate solicit in Relay-Forward message and send to both DHCP servers

   •  Both DHCP servers receive the message - only the elected primary server will respond (if enabled via KEA).

   •  Primary DHCP server sends Relay-Reply to both R1 & R2 with DHCP Offer (Advertise) holding IA_NA and IA_PD information

   •  Client receives DHCP Advertise message from R1 & R2 and returns a Request message to the DHCP server, accepting the offer

   •  R1 & R2 send request via Relay-Forward  packets unicast to the primary DHCP server

   •  Primary DHCP server receives the request and sends a final Relay-Reply to R1 & R2, finalizing the lease with the client

   •  R1 & R2 have been snooping the exchange the whole time, now that it sees the final Relay-Reply message, it will forward the acceptance to the client (Reply message), and install the lease information into its database. Specific for IPv6, it will install a route to the IA_NA prefix, and IA_PD prefix.

   •  Both Juniper routers will now have subscriber information and routes back to the CPE!

Active Leasequery

OK now we have an IPv6 lease on our subscriber CPE! But lets say R1 in our diagram suffers a critical hardware failure, and the subscriber state and DHCP bindings are lost. At first, traffic will failover to R2 as long as you have VRRP/HSRP configured for your link-local addressing. If the first router comes back up and claims primary though - customer traffic destined to this router will be blackholed.

Here's where Active Leasequery (ALQ) finally comes to the rescue 🦸‍♂️! Lets say ALQ is configured between the core routers. Once R1 comes back online, an ALQ TCP session will be created automatically between the two devices. R2 will at this stage synchronize its bindings with R1, restoring the DHCP bindings on the device along with the subscriber database in the process. Below is verification of the session establishment:

The simpler way of confirming ALQ session establishment is to simply run "show dhcpv6 active-leasequery neighbors" - At least, in theory! I'm currently a month deep into a JTAC case with Juniper where we're trying to get this functionality working with a basic configuration. At this stage it's believed to be a bug preventing the feature from working on the MX10003 (version 19.4R1-S2.2). I have a continually breaking ALQ TCP session and none of the Juniper ALQ verification commands actually work️ yet 🤦‍♂️.  

In the next post I will talk about the actual ALQ configuration, session establishment process, and verification commands. This is assuming I can get the feature working with Juniper and it isn't a world ending bug 🤞!