Automating Network Device Backup and Configuration Change Detection with Oxidized and Git

Introduction#

Network infrastructure is the backbone of any modern organization. Managing and maintaining the configurations of network devices like routers, switches, and firewalls is a critical task. Manual configuration management is time-consuming, error-prone, and difficult to scale. This article explores how to automate network device backups and configuration change detection using Oxidized and Git, providing a robust and auditable solution for network management.

Oxidized is a network device configuration backup tool that supports a wide range of vendors and devices. Git is a distributed version control system that provides a powerful mechanism for tracking changes, collaborating, and reverting to previous configurations. Combining these tools enables a streamlined and automated approach to network configuration management.

Why Automate Network Configuration Management?#

Before diving into the implementation, let’s understand the benefits of automating network configuration management:

Reduced Risk of Human Error: Automating backups and change detection minimizes the risk of manual errors during configuration changes.
Improved Compliance: Automated backups provide an audit trail for compliance requirements, demonstrating adherence to security policies.
Faster Recovery Time: In case of device failure or misconfiguration, having readily available backups significantly reduces recovery time.
Enhanced Collaboration: Git allows multiple engineers to collaborate on network configurations, track changes, and resolve conflicts effectively.
Increased Efficiency: Automating routine tasks frees up network engineers to focus on more strategic initiatives.
Configuration Drift Detection: Identify unauthorized or unintentional configuration changes, allowing for quick remediation.

Setting up Oxidized#

First, you need to install and configure Oxidized. Oxidized is typically installed on a dedicated server or virtual machine.

Installation (Example using Debian/Ubuntu):

sudo apt update
sudo apt install ruby ruby-dev build-essential git
sudo gem install oxidized
sudo gem install oxidized-web --no-document #Optional, for web interface

Configuration:

The main configuration file for Oxidized is typically located at ~/.config/oxidized/config. Here’s a sample configuration:

# ~/.config/oxidized/config

username: your_username
password: your_password
model: junos # Example: junos, cisco_ios, etc.
interval: 3600 # Run every hour
threads: 30
debug: false
timeout: 20
retries: 3
prompt: !ruby/regexp /^.*[@>#]\s?$/
rest: 127.0.0.1:8888 #Optional, for web interface

source:
  default: csv
  csv:
    file: ~/.config/oxidized/devices.csv
    delimiter: ","
    map:
      name: 0
      model: 1
      group: 2

input:
  default: ssh
  debug: false

output:
  default: git
  git:
    user: Oxidized
    email: [email protected]
    repo: /opt/oxidized/gitrepo # Location of the Git repository
    branch: main

model_map:
  cisco_ios: ios
  junos: junos
  arista_eos: eos

Explanation:

username and password: Credentials used to connect to network devices. Consider using SSH keys for enhanced security.
model: The device type (e.g., junos, cisco_ios). Oxidized supports a wide variety of device models.
interval: How often Oxidized should run (in seconds). 3600 seconds equals one hour.
source: Specifies where Oxidized gets the list of devices to back up. In this example, it’s a CSV file.
output: Configures where Oxidized stores the backups. Here, it’s configured to use Git.
repo: The path to the Git repository where configurations will be stored.

Device List (devices.csv):

Create a CSV file (e.g., ~/.config/oxidized/devices.csv) containing the list of network devices:

hostname1,cisco_ios,core
hostname2,junos,edge
hostname3,arista_eos,spine

Each line represents a network device, with the hostname, model, and group (optional) separated by commas.

Setting up Git Repository#

Oxidized will store the network device configurations in a Git repository. You need to initialize this repository.

sudo mkdir -p /opt/oxidized/gitrepo
sudo chown -R $(whoami):$(whoami) /opt/oxidized/gitrepo
cd /opt/oxidized/gitrepo
git init --bare

This creates an empty Git repository at /opt/oxidized/gitrepo. The --bare option creates a repository without a working directory, which is suitable for storing backups. Ensure the Oxidized user has write access to this directory.

Running Oxidized#

Start Oxidized:

oxidized

Oxidized will now connect to the devices listed in devices.csv, retrieve their configurations, and store them in the Git repository. You can check the logs for any errors.

Automating Configuration Change Detection#

With Oxidized backing up configurations to a Git repository, you can easily detect configuration changes. Here’s how:

Regularly Check for Changes: You can use a cron job or a similar scheduler to regularly check the Git repository for changes.
Git Commands for Change Detection: Use Git commands to compare the current configuration with the previous version.
```
cd /opt/oxidized/gitrepo
git fetch origin main
git diff origin/main
```
This will show the differences between the current configuration (on the main branch) and the previous version.
Automated Notifications: You can integrate Git hooks or CI/CD pipelines to automatically send notifications (e.g., email, Slack message) when configuration changes are detected. For example, you could use a post-receive hook to trigger a script that sends an email notification.
Example Post-Receive Hook (post-receive):
```
#!/bin/bash

while read oldrev newrev ref
do
  branch=$(git rev-parse --symbolic --abbrev-ref $ref)
  if [ "main" = "$branch" ]; then
    git diff $oldrev..$newrev | mail -s "Network Configuration Change Detected" [email protected]
  fi
done
```
Make the script executable:
```
chmod +x /opt/oxidized/gitrepo/hooks/post-receive
```
This script will send an email to [email protected] whenever a change is pushed to the main branch.

Advanced Configuration and Customization#

Oxidized and Git offer numerous options for advanced configuration and customization:

SSH Keys: Use SSH keys instead of passwords for secure device access.
Device Groups: Organize devices into groups and apply different configurations or backup schedules.
Custom Models: Create custom device models for devices not officially supported by Oxidized.
Web Interface (Oxidized-Web): Use the Oxidized-Web interface to view device configurations and track changes through a web browser.
Integration with CI/CD Pipelines: Integrate Oxidized and Git with CI/CD pipelines for automated configuration testing and deployment.
Secrets Management: Use a secrets management tool (e.g., HashiCorp Vault) to securely store and manage device credentials.

Example: Integrating with Slack#

To send notifications to Slack when configuration changes are detected, you can modify the post-receive hook to use the Slack API.

Install curl (if not already installed):

sudo apt install curl

Modified Post-Receive Hook (post-receive):

#!/bin/bash

SLACK_WEBHOOK_URL="YOUR_SLACK_WEBHOOK_URL"

while read oldrev newrev ref
do
  branch=$(git rev-parse --symbolic --abbrev-ref $ref)
  if [ "main" = "$branch" ]; then
    DIFF=$(git diff $oldrev..$newrev)
    MESSAGE="Network Configuration Change Detected:\n\`\`\`${DIFF}\`\`\`"

    curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"${MESSAGE}\"}" $SLACK_WEBHOOK_URL
  fi
done

Replace YOUR_SLACK_WEBHOOK_URL with your actual Slack webhook URL. You’ll need to create a Slack app and configure an incoming webhook.

This script sends a message to Slack containing the configuration changes.