Getting ElasticSearch Node Versions

Leave a reply

Kibana doesn’t tell you what version your nodes are currently running which can be frustrating if you get distracted during an upgrade process. Here is a simple PowerShell script that gets all the nodes and their current version.

$cred = Get-Credential
$nodes_raw = Invoke-RestMethod -Method Get -Uri "http://elasticsearch:9200/_nodes" -Credential $cred

#get the names of each node
$node_names = $nodes_raw.nodes | Get-Member -MemberType NoteProperty | Select -ExpandProperty Name

#convert from object properties to an array of objects
$nodes = foreach ($node_name in $node_names) {
    $nodes_raw.nodes."$node_name"   
}

# Select desired info about the node
$nodes | select name, version
Advertisement

Grafana Elasticsearch Moving Averages

Leave a reply

Adding moving averages to a graph of Elasticsearch data in grafana is pretty easy, but not well documented

  1. Add a new metric
  2. Choose MovingFunction and pick the metric you want an average on
  3. Expand the options, set Window to the number of samples you want to have your moving function on
  4. Script should be one of the following:
    1. MovingFunctions.max(values)
    2. MovingFunctions.min(values)
    3. MovingFunctions.sum(values)
    4. MovingFunctions.unweightedAvg(values)
    5. MovingFunctions.linearWeightedAvg(values)

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-movfn-aggregation.html#:~:text=The%20Moving%20Function%20aggregation%20allows,script%20in%20the%20values%20variable.

Use PowerShell to export AD User accounts to a CSV with their properties flattened

Leave a reply

This queries for specific users (modify this to match your need)

Taking a list of users Flatten-Object will go through their properties and join multivalued attributes with a semi-colon so that you can export it in a useful way to CSV.

# Change this to and LDAP filter that fits your needs
$admins = Get-ADUser -filter "employeeType -eq 'Administrator Acct'" -Properties *

# take multivalued properties and join them with ;
function Flatten-Object {
    [CmdletBinding()]
    param(
        [Parameter(ValueFromPipeline)]
        $object
    )
    process {
        $flattened_object = [pscustomobject]@{}

        foreach ($field in $object | get-member) {
            if ($field.MemberType -eq “Property” -and $field.Name -notlike “__*”) {
                $flattened_object | Add-Member -NotePropertyName $field.Name -NotePropertyValue $($object.$($field.Name) -join ";")
            }
        }
        $flattened_object
    }
}

$admins | Flatten-Object | Export-Csv -Path "user.csv" -NoTypeInformation

Prettify/Format LDAP Queries using Powershell

Leave a reply

Here is a simple script to indent add new lines to a PowerShell query. 

$ldap = "(&(&(objectCategory=person)(objectClass=user))(lastLogonTimestamp<=128752108510000000)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))"

$ldap = $ldap -replace '\(([\&\|\!]+)', "(`$1`n"
$ldap = $ldap.replace(')', ")`n")

$lines = $ldap.split("`n")
$indent = 0
$new_lines = ForEach ($line in $lines) {
    if  ($line -eq ")") {
        $indent--
    }

    ("`t" * $indent) + $line

    if ($line -eq "(&" -or $line -eq "(|" -or $line -eq "(!") {
        $indent++
    }
}

$new_lines

Results in a nicely formatted query:

(&
	(&
		(objectCategory=person)
		(objectClass=user)
	)
	(lastLogonTimestamp<=128752108510000000)
	(!
		(userAccountControl:1.2.840.113556.1.4.803:=2)
	)
)

Dynamically Getting AD User Properties

Leave a reply

Sometimes you want to return properties that match a specific pattern. The following will do that for you:

Get-ADUser Administrator -Properties * | %{Get-Aduser $_.sAMAccountName -Properties @($_ | Get-Member -MemberType Property | Select -ExpandProperty Name | Where {$_ -like "ms*"})}

This gets a user with all properties, then filters those properties by the final like statement and re-gets the user with just those properties. You can also do it with a single get-aduser:

Get-ADUser Administrator -Properties * | %{$_ | Select -Property @($_ | Get-Member -MemberType Property | Select -ExpandProperty Name | Where {$_ -like "ms*"})}

The only downside to this is it doesn’t include the default attributes, so it can be a bit less usefull. If you want to include some specific attributes + all matching a like statement then you can do the following:

Get-ADUser Administrator -Properties * | %{$_ | Select -Property @(@("Name","samACcountName", "Department", "Description", "DisplayName", "DistinguishedName", "employeeType") + ($_ | Get-Member -MemberType Property | Select -ExpandProperty Name | Where {$_ -like "ms*"}))}

And finaly as a code block instead of a one liner:

$user = Get-ADUser Administrator -Properties * 
$msProperties = $user | Get-Member -MemberType Property | Select -ExpandProperty Name | Where {$_ -like "ms*"}
$user | Select -Property @(@("Department", "Description", "DisplayName", "DistinguishedName", "employeeType") + $msProperties)

Reducing the number of replicas in Elasticsearch

Leave a reply

Sometimes you just want to run a single Elasticsearch node and not have it constantly alert that it has no were to write its replicas. Since Elasticsearch and more templates default to at least 1 replica we have to make changes to Elasticsearch and to the templates. First change the default:

curl -XPUT 'localhost:9200/_settings' -d '
{
    "index" : {
        "number_of_replicas" : 0
    }
}
'

Then we can list all the templates and figure out which ones need updates as well

curl -XGET -H 'Content-Type: application/json' 'localhost:9200/_template/*?pretty'

Then for each of the templates you want to update. For instance to update a Logstash template use the following:

curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/logstash-*/_settings' -d '{ "number_of_replicas" : 0 } }'

Python OAuth2 auth and bearer token caching

Leave a reply

To access many APIs you need to use OAuth2, sending a client id and secret to an endpoint to get a token back. Then send that token with future calls as authentication.

The particular API I was calling would also return a number of seconds the bearer token would be good for.

Hopefully this code will help jumpstart someone else along the way to using python and APIs.

#!/usr/bin/env python3

import requests
import json
import datetime

import os.path
from os import path

cache_file = "/var/tmp/oauth.json"
client_id = '**your client_id here**'
client_secret = '**your secret here**'

def getToken():
    url = "https://api.wherever.com/oauth2/token"

    data = {
      'client_id': client_id,
      'client_secret': client_secret,
      'grant_type': 'client_credentials'
    }

    response = requests.post(url, data=data)

    data = json.loads(response.text)
    expiration_time =  datetime.datetime.now() +  datetime.timedelta(seconds=data['expires_in'])
    data['expiration_date'] =   int(expiration_time.utcnow().timestamp())

    with open(cache_file, "w") as outfile:
        json.dump(data, outfile)
    return data



if path.exists(cache_file):
    #Reading cache
    with open(cache_file, "r") as infile:
        access_token = json.load(infile)
    if int(datetime.datetime.now().timestamp()) > access_token['expiration_date']:
        #Token expired, get new
        acces_token = getToken()
else:
    #No cached value, get and cache
    access_token = getToken()


bearer_token = access_token["access_token"]
headers = {
        'Authorization': f'bearer {bearer_token}'
}

#The rest of the requests go here and pass that header

Use PowerShell to find what process is using a port

Leave a reply

Sometimes you just need to know what ports a process is listening on, or what process is listening on a port…so here you go, just replace the process or port and have at it.

$process = "svcHost"
Get-NetTCPConnection | Where OwningProcess -in (Get-Process | Where ProcessName -eq $process | Select -ExpandProperty Id)

$port = 135
Get-Process -Id (Get-NetTCPConnection -LocalPort $port).OwningProcess

Steps for Setting up Rancid + CentOS + Git

Leave a reply

Setting up Rancid so that it pushes config up into a Git repository. This way you have Rancid getting the config when it changes, and Git storing a history of those changes.

Install CentOS & update

# install pre-requisites
sudo yum install wget gcc perl tcl expect
#setup needed groups and users
sudo groupadd netadm
sudo useradd -g netadm -c "Networking Backups" -d /home/rancid rancid

#working directory to store source
sudo mkdir /home/rancid/tar

#download source and extract
sudo su
cd /home/rancid/tar
wget ftp://ftp.shrubbery.net/pub/rancid/rancid-3.9.tar.gz
tar -zxvf rancid-3.9.tar.gz

# configure/make and install
cd ./rancid-3.9
./configure --prefix=/usr/local/rancid
make install

# copy template config and set file permissions
chmod 0640 /home/rancid/.cloginrc
chown -R rancid:netadm /home/rancid/.cloginrc
chown -R rancid:netadm /usr/local/rancid/
chmod 775 /usr/local/rancid/

Configure Git

  1. yum install git
  2. Modify /usr/local/rancid/etc/rancid.conf
  3. Change RCSSYS=git
  4. Save and close
  5. Switch to rancid user su - rancid
  6. Create SSH key ssh-keygen -o -t rsa -b 4096 -C "email@example.com"
  7. Copy the key to insert into github/gitlab `vim ~/.ssh/id_rsa.pub`
  8. Setup Defaults 
    1. git config --global user.name "Rancid" git config --global user.email "email@example.com"
  9. Configure Rancid Groups
  10. Build initial group folders `​/usr/local/rancid/bin/rancid-cvs`
  11. Go to the device group folder cd /usr/local/rancid/var
  12. For each Device group 
  13. cd {device group} git remote rename origin old-origin git remote add origin git@{git server}:{repo}/{git project}.git git push -u origin --all
  14. Setup hook to push to the git server. In each device group
    1. Open the  post-commit hoof file 
vim .git/hooks/post-commit

#!/bin/sh
# push the local repo to the remote server on commit
git push origin

Finding who is using a Windows Share in PowerShell

Leave a reply

You can find the users and computers connected to a share pretty easily in powershell.

$share_name = "*"
$shares = Get-WmiObject Win32_ServerConnection 
$shares = $shares | Where-Object {$_.ShareName -like $share_name} | `
                    Select-Object ShareName, UserName, `
                                  @{n="Computer";e={[System.Net.Dns]::GetHostEntry($_.ComputerName).HostName}}
$shares | Group-Object -Property ShareName | ` 
          Select Name, `
                 @{n="Computers";e={$_.Group | Select -ExpandProperty Computer | Get-Unique}},`
                 @{n="Users";e={$_.Group | Select -ExpandProperty UserName | Get-Unique}},`
                 Group

$share_name is a filter to narrow down the shares you are checking. Other than that it gets all the share connections then groups them by share. It shows unique users and computers connected to each share.