About this Document

v2.1.2, February, 2016

© 2016 Wedge Networks, Inc. All rights reserved.

No part of this publication including text, examples, diagrams or illustrations may be reproduced, transmitted, or translated in any form or by any means, electronic, mechanical, manual, optical or otherwise, for any purpose, without prior written permission of Wedge Networks, Inc..

Trademarks

Wedge Cloud Network Defense™ is a trademark of Wedge Networks, Inc.. Other product and company names used in this document are used for identification purposes only, may be trademarks of other companies, and are the property of their respective owners.

Wedge Cloud Network Defense™ and associated software are protected by, or for use under, one or more patents.

Contact information

For more information, visit http://www.wedgenetworks.com.

For any errors or omissions in this document, email techdoc@wedgenetworks.com.

For comments or technical issues with any Wedge Networks, Inc. products or services, email support@wedgenetworks.com with the following information:

  • Name

  • Company

  • Email

  • Phone number

  • Product name

  • Product version

  • Product model

  • Description of issue

1. Introduction

1.1. What is Wedge Cloud Network Defense?

pic1
Figure 1. Overview

Wedge Cloud Network Defense delivers the world’s highest-performance cloud security solution for service providers and enterprises which enables world-class security that is non-invasive and low-impact.

Wedge Cloud Network Defense receives user policy and endpoint identification from a number of sources depending on how it is deployed. Identity Management System, Telco OSS/BSS, or registration with Portal are some examples.

The system then uses that information to control the network flow to inspect desired user traffic through its network function virtualization for security (NFV-S) compute stack.

The network function virtualization for security is the foundation of Wedge Cloud Network Defense’s Elastic Security Service Orchestration. Working as a pooled resource, it dynamically applies specific policy-based security inspection to specific user device traffic in response to network load. It is powered by Wedge Networks’ patented WedgeOS which is an embedded operating system that enables the delivery of a variety of security functions as an agnostic, open and easily consumable service.

The Wedge Cloud Network Defense Cloud Conductor coordinates all the cloud operational aspects to dynamically monitor and maintain virtual instances and effectively apply platform resources.

Wedge Cloud Network Defense can be set up in a variety of ways. For more information, see the Deployment Guide.

1.2. Features

pic2
Figure 2. Wedge Cloud Network Defense

Wedge Cloud Network Defense provides the following capabilities:

  • Elastic Security Services Orchestration™ integrates software-defined networking (SDN), network-function virtualization (NFV) and cloud management capabilities to dynamically process high volumes of network traffic in real time without compromising the performance of the world’s largest networks. This feature is only applicable to OpenStack-based cloud infrastructure.

  • Embedded security inspects and applies policies without requiring traffic to leave the network

  • Exhaustive content inspection of all inbound and outbound traffic to defend devices, users, data, and web applications

  • Open platform delivers interoperability with different identity management programs, OSS systems, cloud orchestration systems, SDN controllers and network components

  • Support for standards-based cloud management and virtualization technologies such as OpenStack, KVM, and VMWare

  • Intuitive user interface with a single-pane-of-glass for managing all elastic security services

1.3. Terminology

Cloud Conductor

Coordinates the cloud operational aspects like creating and monitoring the virtual instances required for Wedge Cloud Network Defense, configuring and communicating SDN and virtual instance environment details to Wedge Cloud Network Defense components, and interacting with underlying virtualization solution through the Cloud Controller.

Compute node

Physical hardware configured for a virtualization solution to provide the processing, memory, storage, and networking resources to run multiple virtual instances.

Elastic scaling

If CPU load increases in the primary pool and the high threshold is reached, an NFV-S instance is deployed into the scanning pool to help reduce the overall load. If CPU load decreases and the low threshold is reached, the NFV-S instance is terminated.

Network function virtualization for security (NFV-S)

WedgeOS embedded security services operating as a network function virtualized. The WedgeOS NFV instances are elastically provisioned that is based on inspection load so security can scale to meet demand.

Primary pool

The pool of VNFs that are active and ready to do scanning.

SDN Terminus

Provides control logic by communicating with networking devices residing in the data plane. This SDN control allows precise network traffic to be forwarded to NFV-S instances to provide security services for specific endpoints.

Service Conductor

Manages security policies for registered endpoints and identity management.

Software-defined networking (SDN)

A computer networking architecture that separates the network control plane from the forwarding plane so that each component can be programmable and optimized.

Standby pool

The pool of VNFs that are provisioned to help scan in case the primary pool receives a high amount of traffic that exceeds its threshold.

Virtual instance or virtual machine

An emulation of an operating system (OS) and any software installed on that OS, that runs as an isolated entity managed by a virtualization solution.

Virtual machine image

A virtual instance, which at a selected point of its operation is ‘imaged’ or ‘snapshot’ and stored, so that new virtual instances are started from this stored reference point. Examples include raw, qcow2 (QEMU copy-on-write version 2), ami (amazon machine image), and vmdk (virtual machine disk).

Virtualization solution or hypervisor

Technology that allows multiple virtual OS/software installations to run on a single hardware node, such that each virtual machine has private virtualized hardware (CPU, memory, disk, network interfaces, etc). Examples of virtualization solutions are VMWare, KVM, XEN, etc.

Virtualized network function (VNF)

A component of a network infrastructure that handles specific network functions such as intrusion prevention.

1.4. Hardware requirements

Ensure that your hardware meets the following requirements.

  • 4 devices or nodes

  • Controller node (1)

    • 4 CPU cores

    • 8 GB RAM

    • 120 GB HDD

    • 4X 1GB NIC

  • Open vSwitch (1)

    • 4 CPU cores

    • 8 GB RAM

    • 120 GB HDD

    • 8X 1GB NIC

  • Compute node (2)

    • 28 CPU cores

    • 48 GB RAM

    • 500 GB SSD

    • 4X 1GB NIC

Sizing considerations

  • WedgeOS instances require 2 vCPUs each

  • Service Conductor requires 2 vCPUs

  • Cloud Conductor requires 2 vCPUs

  • SDN Terminus requires 4 vCPUs

2. Components overview

2.1. Cloud Controller

The Cloud Controller is based on OpenStack technology and provides the following:

  • Repository of virtual instances and an API to query and control these instances in the virtualization solution. The virtual instances are WedgeOS, Wedge Cloud Conductor, Wedge SDN Terminus, and Wedge Service Conductor.

  • Repository of virtual images required for Wedge Cloud Defense operation. Images can include WedgeOS and Ubuntu.

  • Creation of multiple virtual networks with IP/subnet allocations for software-defined networking.

pic4
Figure 3. Cloud Controller

2.2. Cloud Conductor

pic5
Figure 4. Cloud Conductor

The Cloud Conductor provides health monitoring of virtual instances, creates and destroys NFV-S instances based on the inspection load. It provides management of the NFV-S instances, SDN Terminus, and OpenFlow Switch configuration.

It coordinates cloud computing abilities. A single instance controls all cloud computing operations.

As illustrated in the diagram, the Cloud Conductor has the following components:

  • The Wedge Instance Monitor monitors the health of NFV-S instances and provides elastic scaling based on inspection load.

  • The REST API provides remote management and meta data to populate the Cloud Conductor Dashboard.

  • Service Handlers provides business logic to communicate with various the cloud components and the operation of the Cloud Conductor.

  • Pool Managers track and manage NFV-S instances, SDN Terminus, and OpenFlow switch configuration.

Cloud Conductor dashboard

pic6
Figure 5. Cloud Conductor dashboard

The Overview page provides real-time graphs on the resource usage of a pool of NFV-S instances or the scanning pool.

The Throughput page provides bandwidth usage for the Ingress, Egress, and Control networks.

The Infrastructure page provides a read-only view of the Quota Limits in the Cloud Controller (OpenStack), flavors that have been configured, and images that have been loaded.

The Configuration page provides a read-write view of the configuration options for the Instance Monitor and the Cloud Controller.

pic7
Figure 6. CPU Usage

The CPU Usage graph shows the average CPU usage of all the active NFV-S instances in the primary scanning pool. This time series graph updates periodically as the CPU load changes over time in the primary pool. If CPU load increases and the high threshold is reached, a NFV-S instance is deployed into the standby pool to help reduce the overall load. If CPU load decreases and the low threshold is reached, a NFV-S instance is terminated and resources are returned to the virtualization solution.

pic8
Figure 7. Instance Count

The Instance Count graph shows the number of NFV-S instances in the primary scanning pool. The series graph updates periodically to show how instances are adding and removed from the primary scanning pool over time.

2.3. Service Conductor

pic9
Figure 8. Service Conductor

As illustrated in the diagram, the Service Conductor has the following components:

  • Service handlers distribute SDN-S policies to the SDN Terminus to identify endpoint user traffic to be scanned by virtualized network functions.

  • The REST API manages subscriber registration and policy management.

  • The Smartfilter service manages user security policy configuration and identity management.

  • The Database service contains registered subscribers and service policies for the subscribers.

  • The Central syslog service is a centralized logging of security events from virtualized network function instances.

2.4. SDN Terminus

pic10
Figure 9. Terminus

The SDN Terminus receives subscriber policy and endpoint identification from the Service Conductor. It receives NFV-S instance accounting from the Cloud Conductor. It controls the data plane to flow network traffic to appropriate NFV-S instance for security inspection.

It provides SDN-S control on the data plane. One instance is configured for each OpenFlow switch.

As illustrated in the diagram, the SDN Terminus has the following components:

  • The REST API manages the identity of subscriber endpoints, policies, and NFV-S that is provided to the SDN Terminus to direct the load balancing logic.

  • The Cache Managers views the flow reporting of current flows that are directed to the Wedge Cloud and the policy pool for each.

  • The OpenFlow Controller manages the OpenFlow switch to forward subscriber traffic from data plane to the security virtualized network function. It serves as a Load Balancer to distribute security scanning tasks to the network function virtualized instances.

  • The Database service contains SDN policies and NFV-S instances.

2.5. Network function virtualization for security (NFV-S)

The network function virtualization for security fetches policy and endpoint identification from the Service Conductor. It provides the security inspection and enforcement of data plane traffic and reports security events to WedgeIQ for dashboard display, reporting, analytics, and forensics.

The security functions are provided by the pool of managed security VNFs. Many VNFs are security applications that are running on the award winning WedgeOS™. Third-party VNFs can also be orchestrated to make their functions available to the NFV-S.

2.6. Customer portal

pic11
Figure 10. Customer portal

Wedge Cloud Network Defense comes with a reference self-service customer portal. Use the self-service Customer portal to register and enable security scanning services to protect endpoints that connects to the Wedge Cloud Network Defense.

pic19
Figure 11. Customer portal login and registration

Clients can register an account. For new accounts, an email is auto-generated and sent to the client with instructions on how to connect the endpoint device to Wedge Cloud Network Defense by using a VPN connection.

The following are the available services in the portal:

  • Anti-Virus

  • Anti-Spam

  • Data Loss Prevention (DLP)

  • WebFilter

  • URL Access

Additional configuration is required for DLP, WebFilter, and URL Access services.

pic20
Figure 12. Customer portal overview

Security Services

The following is a list of security services that can be provisioned by the customer portal. Contact Wedge support if you would like to have other security services included.

Anti-Virus Service: scans HTTP, FTP, and e-mail protocol traffic for known viruses and malware.

Anti-Spam Services: scans e-mail protocol traffic for spam.

Content Control Services

Data Loss Prevention (DLP): scans HTTP and e-mail protocol traffic for specified keyword patterns or custom keywords.

Keyword patterns available: Credit Card, SSN (USA), and SIN (Canada)

WebFilter: block HTTP protocol traffic based on website categories selected.

URL Access: configure URL pattern to match against HTTP protocol traffic. Enforce strict safe search with Google, Yahoo!, and Bing search engines.

2.6.1. Enabling Anti-Virus

  1. Once logged into Self Service Customer Portal, select Anti-Virus service from the Available Services panel and drag over to Selected Services panel.

pic21
Figure 13. Select Anti-Virus

  1. Once the Anti-Virus service is in the Selected Services panel, the Anti-Malware configuration dialog will appear in the Configure Your Services panel and the service will be disabled by default.

pic22
Figure 14. Anti-Virus

  1. Click Enable to start Anti-Virus protection for endpoint devices.

pic23
Figure 15. Enable Anti-Virus

End user experience

pic24
Figure 16. Anti-Virus

2.6.2. Enabling Anti-Spam

  1. Once logged into Self Service Customer Portal, select Anti-Spam service from Available Services panel and drag over to Selected Services panel.

pic25
Figure 17. Select Anti-Spam

  1. Once the Anti-Spam service is in the Selected Services panel, the Anti-Spam configuration dialog will appear in the Configure Your Services panel and the service will be disabled by default.

pic26
Figure 18. Anti-Spam

  1. Click Enable to start Anti-Spam protection for endpoint devices.

pic27
Figure 19. Enable Anti-Spam
pic28
Figure 20. Anti-Spam end user experience

2.6.3. Enabling Data Loss Prevention

  1. Once logged into Self Service Customer Portal, select Data Loss Prevention service from Available Services panel and drag over to Selected Services panel.

pic29
Figure 21. Select Data Loss Prevention

  1. Once DLP service is in the Selected Services panel, the Data Loss Prevention configuration dialog will appear in the Configure Your Services panel and will be disabled by default. Configure the traffic to scan for specified keyword patterns or enter individual custom keywords.

pic30
Figure 22. Data Loss Prevention

  1. Click Enable to start Data Loss Prevention protection for endpoint devices.

pic31
Figure 23. Enable Data Loss Prevention

After enabling, the DLP policy configuration prevented the e-mail message and attachments from being sent due to verdict from scan.

pic31
Figure 24. Data Loss Prevention end user experience

2.6.4. Enabling WebFilter

  1. Once logged into Self Service Customer Portal, select the WebFilter service from the Available Services panel and drag over to the Selected Services panel.

pic33
Figure 25. Select WebFilter

  1. Once the WebFilter service is in the Selected Services panel, the WebFilter configuration dialog will appear in the Configure Your Services panel and will be disabled by default. Select one or more website categories to be blocked from HTTP traffic. There are over 100 categories to choose from.

pic34
Figure 26. WebFilter

  1. Click Enable to start WebFilter protection for endpoint devices.

pic35
Figure 27. Enable WebFilter

After enabling, the block page is displayed based on the WebFilter category that matched the Client HTTP request and the action taken based on the verdict from policy configuration.

pic36
Figure 28. WebFilter end user experience

2.6.5. Enabling URL Access

  1. Once logged into Self Service Customer Portal, select the WebFilter service from the Available Services panel and drag over to the Selected Services panel.

pic37
Figure 29. Select URL Access

  1. Once URL Access service is in the Selected Services panel, the URL Access configuration dialog will appear in the Configure Your Services panel and will be disabled by default. Configure the URL patterns to be blocked from HTTP traffic. You can also configure strict safe search with the specified search engines.

pic38
Figure 30. URL Access

  1. Click Enable to start URL Access protection for endpoint devices.

pic39
Figure 31. Enable URL Access

2.6.6. Integrating with a third-party portal

Wedge Cloud Network Defense can integrate with a third-party portal to register or identify subscribers and endpoint devices. The portal must certify and adhere to the Service Conductor REST API specification.

The portal must access the control network so that it can communicate with the Wedge Service Conductor, which is also in the control network.

The portal loads the available security services that are licensed. Currently, it has the following five static services:

  • Anti-virus

  • Anti-spam

  • URL Access

  • WebFilter

  • Data Loss Prevention

Service categories are also loaded from the Wedge Service Conductor.

The self-service portal sends the following operational data to the Wedge Service Conductor.

  • registration of subscriber

  • policy configuration for subscriber based on the services that have signed up

3. Administering

3.1. Managing the OpenStack dashboard

The OpenStack dashboard can be accessed by using the URL http://[controller-ip]/horizon

Refer to install.properties in wedgecnd-controller package for authentication details to access dashboard. The account provides access to Project views and operations and Admin views and operations.

The Project can be considered access to the Tenant, whereas Admin is for OpenStack as a whole.

pic40
Figure 32. OpenStack Dashboard

Health and instance overview

pic41
Figure 33. Health and instance overview

The cloud health overview can be viewed from Project > Compute > Overview.

This page provides the Limit Summary on how much of the quota is used by the Tenant based on various resources.

The Usage table lists the Instances that have been deployed for the Tenant and how much resources each instance is using.

The instance overview can be viewed from Project > Compute > Instances.

The Instance details are the Name, IP address, Image, Flavor, Status, and Uptime.

The Actions are Launch Instance, Reboot Instance, Terminate Instance, Create Snapshot, and more.

Instance details overview

pic42
Figure 34. Instance details overview

From the Projects > Compute > Instances page, click on the Instance name and the Instance Details page will load.

The instance details page is split up into Overview, Log, and Console tabs.

Overview tab shows instance meta-data, resources used by the instance and IP addresses of the various virtual networks it is connected to.

Log tab shows the console log when the instance booted.

The Console tab provides the means to console into the instance for remote management. For some browsers, you may have to click the link to only show the console for keyboard inputs to work.

Note Console access is only available for NFV-S instances. Any instance that is launched with the wedgecnd_key.pem will have their console unaccessible as the key needs to be used to remotely access the instance.

Rebooting and terminating an instance

pic43
Figure 35. Reboot and terminate instances

To reboot or terminate an instance from the cloud, navigate to Project > Compute > Instances page.

  1. Select the instance to perform action on.

  2. To reboot, click Soft Reboot Instance.

If soft reboot action was unsuccessful, try Hard Reboot Instance under More.

  1. To terminate, click Terminate Instance.

Creating a snapshot

During the operation of Wedge Cloud Network Defense, there will be a need to create a snapshot of a NFV instance and to have newer NFV instances that are created to boot based on this snapshot.

  1. Navigate to Projects > Compute > Instances page. Find the NFV instance to create a snapshot from and click Create Snapshot button.

pic44
Figure 36. Create Snapshot

  1. Input a name for the snapshot and click Create Snapshot.

pic45
Figure 37. Snapshot name

  1. You will be redirected to the Projects > Compute > Images page and the snapshot will be queued for processing.

pic46
Figure 38. Images page

  1. Once the snapshot has been completed, the status will appear as Active. The snapshot is now ready to be used. The operator can now launch instances based on this snapshot, edit the snapshot details, and delete the snapshot.

pic47
Figure 39. Active status

Adjusting tenant quota limits

During the operation of Wedge Cloud Network Defense, as new compute nodes are added or removed to and from the cloud, the quota limits for the tenant will need to be adjusted accordingly.

  1. Navigate to Admin > Identity Panel > Projects and find the admin project. Under Actions > More, click on Modify Quotas.

pic48
Figure 40. Modify Quotas

  1. There are 13 different quota limits that can be adjusted for the Tenant. When compute nodes are added/removed, the following limits need to be changed:

    • VCPU

    • RAM

    • Instances

    • Ports

The Instances represents the total number of instances (NFV, WedgeCC, WedgeSC, SDN Terminus) that can be launched to the cloud.

The Ports represent the total number of access ports or interfaces that can be created across all instances deployed to the cloud. Once quota limits have been adjusted, click Save.

3.2. Managing the Cloud Conductor dashboard

The Cloud Conductor dashboard can be access by using the URL http://[cloud-conductor-ip]:3000/. This needs to be part of Control Network or have routing to Control Network to access dashboard.

The Overview page provides some basic actions to perform during cloud operation such as launching and terminating an NFV Instance.

The Infrastructure page provides an overview of the quota limits used based on total available.

The Configuration page provides the means to update OpenStack configuration in Wedge Cloud Conductor and adjust Instance Monitor parameters.

Actions

pic49
Figure 41. Instances

The Cloud Conductor dashboard provides the ability to launch a NFV instance to the scanning pool or standby pool.

Click Launch WedgeOS NFV, input a name and select the pool to launch the instance and click Launch.

The NFV instance will be created based on the specified Flavor and Image.

There are two actions that can be performed for NFV instances only:

  • Terminate Instance: Requests the wedgecc-rest-api to terminate instance from the cloud. The difference between terminate instance called through this dashboard versus the Terminate Instance called from OpenStack Dashboard is this will wait for proxy activity to finish before sending the request to nova-api to terminate the instance from the cloud. Terminiate Instance called through OpenStack will terminate the instance immediately.

  • Instance UI: Opens up the NFV Manager dashboard in a separate browser window or tab.

Quota Limits

pic50
Figure 42. Quota Limits

The Quota Limits tab provides a read-only view of the limits currently used compared to max available for the limits exposed by OpenStack.

At this time, quota limits can only be adjusted through the OpenStack dashboard. Operators should monitor quota limits during peak times to observe if limits are being reached. Additional compute nodes could be required if quota limits are being reached on resources such as Instances, VCPU, or Memory.

Configuration

pic51
Figure 43. Configuration

Use the configuration page to update OpenStack parameters and Instance Monitor parameters which control the operation of the Cloud Conductor.

There are some conditions where the following OpenStack parameters will need to be adjusted:

  • Default Image ID: need to update the default image ID if Operator creates a snapshot of an instance and that snapshot should be used for instance creation.

  • Default Flavor ID: need to update the default flavor ID if Operator creates a new flavor to be used when creating instances.

There are some conditions where the following Instance Monitor parameters will need to be adjusted:

  • CPU High Threshold: the high cpu utilization threshold, if reached will start the Scale-Up Task.

  • CPU Low Threshold: the low cpu utilization threshold, if reached will start the Scale-Down Task.

  • Auto-deploy Instance Count: the number of NFV instances to launch into the scanning pool during initialization, and the minimum number of instances to maintain during operation.

  • Standby Pool Min Instance: the number of NFV instances to maintain in the standby pool for each scanning pool.

Once all parameters have been adjusted, click Save.

3.3. Managing remote access

OpenSSH server is installed on all Wedge Cloud Network Defense instances and physical hardware for remote management.

Management of the Instances and physical nodes can be accessed using the Control Network created during installation time.

The Operator must have a connection to this control network to be able to remotely connect to the instances and physical nodes.

During the cloud controller installation, a SSH key is created and added to OpenStack nova-api. This key is then used to boot the WedgeCC, WedgeSC, and SDN Terminus instances. Once the instances are created, the SSH key will need to be used to access those instances via SSH.

The SSH key is not required to access the OpenFlow Switch node, Controller Node, Compute Nodes, or NFV-S instances.

SSH access

To remotely access the WedgeCC, WedgeSC, and SDN Terminus instances, ssh from a terminal to the Cloud Controller node:

% ssh ubuntu@controller-ip-addr

After SSH connection has been established to the controller node, find the private key file wedgecnd-key.pem in the home directory as this will be required to access one of the instances.

% sudo ssh -i wedgecnd-key.pem
 ubuntu@instance-ip-addr

sudo is required as the wedgecnd-key.pem will be owned by root user.

Cloud package upgrades

The Wedge Cloud Network Defense components (Cloud Conductor, Service Conductor, and SDN Terminus) are all installed on Ubuntu Server linux distribution.

For each package, a debian file (.deb) is generated to facilitate the installation of the various cloud components and running additional scripts before/after install or upgrade. As the Wedge Cloud Network Defense components evolve over time, there will be a need to upgrade some of these packages (new features, bug fixes, etc…​)

To upgrade a debian package:

  1. Copy new debian (deb) package onto the Cloud Controller node. This can be achieved with SCP or putting the debian package on a media (CD/USB) and mounting the media onto the controller node.

  2. From controller node, use scp to copy the debian package to the instance you want to upgrade.

    % sudo scp -i wedgecnd-key.pem wedgecc-2.0.1-xx.deb ubuntu@wedgecc-instance-ip-addr

  1. SSH to instance and install the new package.

    % sudo ssh -i wedgecnd-key.pem  ubuntu@wedgecc-instance-ip-addr
% sudo dpkg -i wedgecc-2.0.1-xx.deb

3.4. Managing REST API

REST stands for Representational State Transfer. It uses HTTP protocol as the means of communication between client-server. It uses HTTP verbs (GET,POST,PUT,DELTE) as the action (CRUD operation) to perform for the request.

The content-type (the data payload) used is JSON or JavaScript Object Notation.

The following WedgeCND components have REST API for remote management:

  • Cloud Controller (OpenStack)

  • Cloud Conductor

  • Service Conductor

  • SDN Terminus

You must have a REST client to be able to interface with REST API such as Google Chrome or a stand-alone application.

The wedgesc-rest-api web application is deployed onto Tomcat server. The base URL is http://[service-conductor-ip]:8080/wedgesc/rest.

REST API provides the follow resources:

  • Categories: expose service categories

  • Policy: expose and manage policy configuration for subscribers.

  • Subscriber: expose and manage subscribers registered for security services.

  • Terminus Policy: reload SDN policy configuration to Terminus

The wedgecc-rest-api web application deployed onto Tomcat server. The base URL is http://[cloud-conductor-ip]:8080/wedgecc/rest.

REST API provides the following resources:

  • Action: proxy action requests from cloud conductor dashboard.

  • Cloud Infrastructure: expose quota limits, flavors, and images to cloud conductor dashboard.

  • Form: proxy action request to launch NFV instance and manage OpenStack and Instance Monitor configuration.

  • Health: expose NFV pool resource usage details to cloud conductor dashboard.

  • Instances: expose NFV instance details to cloud conductor dashboard.

  • OpenFlow Switch: expose and manage OpenFlow switch-port mapping.

  • SDN Terminus: expose and manage Terminus pool details.

  • WedgeOS: expose and manage NFV pool details.

The wedge-terminus-rest-api web application is deployed onto Tomcat server. The base URL is http://[terminus-ip]:8080/wedgesc/rest.

REST API has one main resource which provides the following operations:

  • Expose and manage SDN policy configuration

  • Expose and manage NFV instances used by load balancer

4. Maintaining

4.1. Maintaining the compute node

Compute nodes can be added dynamically to the cloud for additional processing abilities.

  1. Access the controller by using secure shell.

  2. Add or update the compute node parameters in install.properties, keeping in mind to set the correct index. The index should be ordered by the following:

    • Compute.node.count - specifies the number of compute nodes in the cloud. This value is not explicitly used in this operation, but if the WedgeCND instances are relaunched, this needs to be updated accordingly

    • Compute.node.hostname.# - the hostname of the compute node identified by this index

    • Compute.node.switch.port.ingress.# - the port on the Open vSwitch that this compute node is connected to for the ingress data network

    • Compute.node.switch.port.egress.# - the port on the Open vSwitch that this compute node is connected to for the egress data network

  1. To add a compute node, run the script:

add_compute_node.sh –i <index of compute node> -s <index of switch>

The index corresponds to the # in the install.properties.

  1. To delete a compute node, run the script:

delete_compute_node.sh –i <index of compute node> -s <index of switch>

The index is the same value as the one specified above in the add operation.

4.2. Enabling cloud recovery

Complete the following steps if you need to re-initialize a Wedge Cloud component to the starting state.

  1. Access the Horizon dashboard (https://<cloud–controller-ip>/horizon).

  2. Delete all instances from the Project > Instances section.

  3. SSH to the controller and navigate to the scripts directory.

  4. Run the deploy_instances.sh script to redeploy the base image for the SDN Terminus, Cloud Conductor, and Service Conductor.

  5. Run install_wedge_cloud.sh to redeploy the components. NFV instances will be automatically deployed.

4.3. Updating the NFV-S image

If the NFV-S instance has been configured with custom settings, some of the settings are not propagated to all instances by using Configuration Sync.

To allow elastic instances to mirror the settings, a snapshot should be created based on the running instance and used to replace the NFV-S image. Currently, the process requires redeploying the cloud in order to properly synchronize all NFV Orchestration components. Future enhancements will be able to avoid this process.

  1. Access the Controller dashboard (http://<cloud–controller-ip>/horizon).

  2. Navigate to the Project > Instances section.

  3. On the right side of the table, select the dropdown box for the NFV-S instance and click Create Snapshot.

  4. For the name, enter the name of NFV-S.

  5. Once the process is completed, navigate to the Project→Images section.

  6. Two NFV-S images should be present in the list. Select the image that is marked as Image and delete the image. If you are replacing an old snapshot, both will be marked as Snapshot, so remove the older entry which usually is lower in the list.

pici12
Figure 44. Updating the NFV-S image

  1. Access the Controller dashboard (https://<cloud–controller-ip>/horizon).

  2. Delete all the NFV-S instances corresponding to this type of NFV-S from the Project→Instances section.

  3. Using the WedgeCC REST API, delete the WedgeOS instances.

  4. Navigate to the scripts directory on the controller by using a shell.

  5. Run the configure_wedgeos_properties.sh script. This will update the image information in WedgeCC and relaunch the instances as necessary.

4.4. Modifying security groups

4.4.1. Modifying rules

The default security group rules that are added during installation may prevent access to the cloud instances. This is because the rule that is created to allow access to instances limits source IPs to come from the external management network.

To access nodes outside of the management subnet which are still able to route to the internal control network, create additional security group rules.

  1. Access the dashboard from http://<CONTROLLER_IP>/horizon and login.

  2. In the left panel click on Project > Compute > Access & Security.

From here there are a few different rules you may want to add.

To allow SSH access to instances:

  1. Click on the Manage Rules action for the common-sg security group.

  2. Click Add Rule.

  3. Select SSH from the Rule dropdown box.

  4. Set Remote to CIDR.

  5. Set the CIDR to the network subnet which requires SSH access to instances.

  6. Click Add.

To allow access to WedgeCC UI:

  1. Click on the Manage Rules action for the wedgecnd-sg security group.

  2. Click Add Rule.

  3. Select Custom TCP Rule from the Rule dropdown box.

  4. Set Direction to Ingress.

  5. Set Port to 3000.

  6. Set Remote to CIDR.

  7. Set the CIDR to the network subnet which requires WedgeCC web UI access.

  8. Click Add.

  9. Repeat steps 1-8 but use port 8989 instead.

To allow access to WedgeCND instance REST API, repeat the previous steps but use port 8080.

4.4.2. Security groups summary

Security groups are useful for providing instance-level rules for restricting communication. Without any rules, all communication is blocked.

Additional rules give the ability to open traffic in the ingress or egress direction on an Instance-based on protocol, port, CIDR.

The security groups are divided into four groups:

wedgeos-data-sg: WedgeOS data port rules
Allow all traffic in both directions so that everything is proxied

wedgeos-control-sg: WedgeOS control port rules
Allow HTTPS access to WedgeCND instances. All WedgeOS control ports have access to required ports on WedgeSC (mongodb, rsyslog, rsync).

wedgecnd-sg: WedgeCND instance (wedgesc, wedgecc, terminus) rules
Allow access to relevant ports to hosts on the external management network. Since the office network is not in the same subnet as the management network, a rule must be manually added to allow connections from the office network.

This includes the ports required for Wedge CC UI, Terminus, and Controller. Allow access to WedgeSC services only to the WedgeOS control group.

Current list of ports:

rsyslog (514 UDP)
rsync (587)
wedgeccui (3000, 8989)
terminus (6633)
REST API (8080)
mongodb (27017)

common-sg: Common rules (shared by WedgeCND instances and WedgeOS control port)
Allow all egress traffic for Internet access. Allow SSH and HTTPS access coming from management network CIDR.

The default security group quota is set to 10, so the rules are combined for all WedgeCND instances into one group to leave room for any more. There is also a quota of 100 for total security group rules per tenant. At this time we use around 25 rules to setup the access as described above.

To make changes to the default rules setup during installation, the configure_security.sh script may be modified to create additional rules. You are basically limited to adding further access to ports for a certain protocol.

Another useful feature is to allow access to instances which are a member of a group. An example of this is with the WedgeCND rule, where access to the mongodb port 27017 is granted to all members of the wedgeos-control-sg group.

The firewall driver dynamically looks up which neutron ports are part of the wedgeos-control-sg group and generates an iptables rule for each instance to allow access to port 27017.

4.5. WedgeOS auto-scale tuning

NFV Orchestration allows the ability to customize the auto-scaling behavior. The preset values will work for general setups, but these values can be modified prior to launching the orchestration comonents.

These properties are located in the wedgeos.properties under the /conf directory on the controller. These values are also configurable through the Cloud Conductor UI.

These should only be modified with the guidance of your Wedge support team.

pici13
Figure 45. Cloud Conductor auto-scale configuration
screen48
Figure 46. Auto-scale configuration

5. Troubleshooting

5.1. Network connectivity

Ensure that the network cables between OpenFlow node to Controller node and compute nodes are connected to each other and can ping each other.

For OpenFlow switch, ensure that the network interfaces are properly identified and connected as shown in the diagrams.

Ensure that the management network can reach the Internet for package downloads during installation and signature updates during operation.

5.2. OpenStack components

Nova is the compute service that facilitates the launching and termination of instances.

  • nova-api

  • nova-scheduler

  • nova-conductor

  • % ps aux | grep nova

Keystone is the identity service that facilitates authentication within OpenStack.

A token is required to make OS API calls.

keystone-all

Glance is the image service which maintains a repository of VM images used to launch instances.

  • glance-api

  • glance-registry

Neutron is the networking service that creates the 3 SDN for Ingress, Egress, and Control.

  • neutron-server

  • % ps aux | grep neutron

Ceilometer is the telemetry service that tracks resource usage of instances (cpu util, bw util).

  • ceilometer-api

  • Ceilometer-collector

  • % ps aux | grep ceilometer

When viewing or modifying configuration files, or looking through log files, enter root mode.

% sudo bash

The configuration files for OpenStack components will be found in /etc/{component-name}

/etc/ceilometer /etc/nova

The log directory for OpenStack components will be found in /var/log/{component-name}.

Each directory could have multiple log files depending on how many services (processes) are running for that component.

/var/log/ceilometer
/var/log/nova

5.3. Wedge Cloud Network Defense components

To troubleshoot Wedge Cloud Network Defense components, SSH to controller node and use ssh key to remotely access instance.

% ssh -i wedgecnd_key.pem ubuntu@instance_ip

The WedgeCC, WedgeSC, and Wedge SDN Terminus are all web applications that are deployed onto a Tomcat application server.

Tomcat log files can be found in:

% cd /var/log/tomcat
% less catalina.out

catalina.out is the main standard out log file. Sometimes if an error has occurred the stack trace may be put to standard error log file, which will be in the format:

% less localhost.[YYYY-mm-dd].log

5.4. OpenFlow switch

  1. Enter root mode to troubleshoot openvswitch.

    % sudo bash

  2. To check the status of the openvswitch bridge, issue the following command:

    % ovs-vsctl show

  3. To check the flows in openvswitch bridge, issue the following command:

    % ovs-ofctl dump-flows ovsbr0

5.5. Frequently asked questions

  • What do I do if an instance is in error state?

SSH to controller node and go through nova-* logs in /var/log/nova to see if any error messages or stack traces shown.

  • Has OpenStack reached a quota limit?

SSH to Cloud Conductor through the ssh key from controller and check catalina.out for any errors that may indicate why instance failed to deploy.

  • Why does the Cloud Conductor dashboard doesn’t show any CPU utilization in the chart?

Check catalina.out on Cloud Conductor instance to see if CPU util is being stored correctly to the RRD file.

If log is showing an average cpu util of 0, then chances are there is an issue with the ceilometer service storing resource usage.

SSH to compute node and restart ceilometer-agent-compute service.

% sudo service ceilometer-agent-compute restart

SSH to controller node and restart ceilometer-collector service

% sudo service ceilometer-collector restart

Continue to monitor catalina.out logs for a few minutes to see if cpu util changes to a non-zero value

  • When testing with EICAR virus, it was not blocked. The anti-virus security service was enabled through the portal.

Check wedgesc-rest-api by using REST client to make sure policy was stored correctly.

Check terminus-rest-api by using REST client to make sure policy was pushed to component.

Check if NFV scanning pool is registered for load balancer.

SSH to OpenFlow switch and dump flows.

6. References

6.1. Wedge Instance Monitor elastic scaling activity

pic12
Figure 47. Part 1
pic13
Figure 48. Part 2
pic14
Figure 49. Part 3

6.2. Use cases

pic15
Figure 50. Use case 1
pic16
Figure 51. Use case 2
pic17
Figure 52. Use case 3
pic18
Figure 53. Use case 4

© 2016 Wedge Networks, Inc.

http://www.wedgenetworks.com

Phone

+1 (403) 276-5358
Fax

+1 (403) 276-5568

support@wedgenetworks.com