Transcript cloudstack-usage-chandra

Cloud Usage with CloudStack
Aug 2014
Chandra Prakash Mishra
© 2014 Sungard AS. All rights reserved.
In next 45 minutes …
Agenda
•
CloudStack Usages
•
Cloudstack configuration to enable Usage Capture
•
Usage Data Generation Process
•
Usage Type
•
Usage Event
•
Usage Records
•
How can Access Usage Data
•
Use Cases
2
Usage Requirements
•
Meter their cloud's consumption and monetize the cloud offerings by charging
for usage
•
Report on the cloud’s resource usage for monitoring/capacity planning
•
Existing billing system, just need access to the usage data
•
Organizations tend to have different billing criteria/charging models and require
access to usage data for their specific needs
Footer – Apply across document
3
Usage in CloudStack today
•
•
•
CloudStack does not offer any native usage monitoring or billing capability
Root Admin can view current System Capacity
Domain Admin and Users can view VM statistics
Footer – Apply across document
4
Usage Server
• CloudStack includes a Usage Server that create summary usage records for the
various resources consumes in cloudstack.
• The usage server is an optional, separately installed component called “cloudstackusage”.
• The usage records are stored in a separate database called “cloud_usage”.
• Its Runs once a day by default .It can be configured to run multiple times per
day and it’s behavior is controlled by the following global configuration
settings.
enable.usage.server
: Whether the usage is active ?
usage.stats.job.aggregation.range : Usage is aggregated for specified number of minutes. e.g default:1440
usage.stats.job.exec.time
: Time when the usage server will start processing. Default: 00.15
usage.execution.timezone
: The timezone to use for usage.stats.job.exec.time. Default: timezone of the management
server
usage.aggregation.timezone
: The timezone to use for Cloudstack usage stats aggregation (default GMT)
Footer – Apply across document
5
CloudStack Event
• Usage events are stored in usage_event table. Any resource for which usage has to be
tracked needs to log usage events when the the resource is created/allocated and when its
deleted/released.
• Network usage is the only exception which doesn’t use usage events. Once these events are
logged usage server will process them and generate Cloudstack usage records.
Description of various columns in usage_event table:
•
•
•
•
•
•
•
•
•
•
type – Event type as described in cloudstack-oss/api/src/com/cloud/event/EventTypes.java
account_id – Id of the account which owns the resource
zone_id – Data center to which this resource belongs to
resource_id – Id of the resource. For VM usage this will be VmId, for template templateId etc
resource_name – Name of the resource.
offering_id – Id of Service Offering in case of Vm usage, Disk offering in case of volume usage etc..
template_id – Id of the template used for a volume / Vm.
size – Size of the resource. Volume Size or Template size etc.
resource_type - Additional type attribute for a resource. Could be used for hypervisor_type etc.
processed – Is set to 0 by default. This field is updated to 1 when the usage server completes processing of this event.
Footer – Apply across document
6
Usage Record Generation
The usage server schedule a job at the time specified by usage.stats.job.exec.time parameter .This
job query cloud databases cloud.usage_event table and generate usage records in the cloud_usage
database table cloud_usage as follows
•
CloudStack events have been logged into cloud.usage_event table .
•
The cloudstack_usage job gets a list of the latest usage events.
•
Inserts these events into cloud_usage.usage_events table.
•
Parses cloud_usage.usage_event and populates Helper tables .
•
Helper table data used to populate cloud_usage.cloud_usage table with aggregation range wise
data .
Footer – Apply across document
7
Usage Type
Type
Type Name
id
Description
6
Tracks the total running time of a VM per usage record period. If the VM is upgraded
during the usage period, you will get a separate Usage Record for the new upgraded VM.
Tracks the total time the VM has been created to the time when it has been destroyed. This
ALLOCATED_VM
usage type is also useful in determining usage for specific templates such as Windowsbased templates.
IP_ADDRESS
Tracks the public IP address owned by the account.
Tracks the total number of bytes sent by all the VMs for an account. Cloud.com does not
NETWORK_BYTES_SENT
currently track network traffic per VM.
NETWORK_BYTES_RECEIV Tracks the total number of bytes received by all the VMs for an account. Cloud.com does
ED
not currently track network traffic per VM.
VOLUME
Tracks the total time a disk volume has been created to the time when it has been destroyed.
7
TEMPLATE
8
ISO
1
2
3
4
5
9
11
12
13
14
RUNNING_VM
Tracks the total time a template (either created from a snapshot or uploaded to the cloud)
has been created to the time it has been destroyed. The size of the template is also returned.
Tracks the total time an ISO has been uploaded to the time it has been removed from the
cloud. The size of the ISO is also returned.
Tracks the total time from when a snapshot has been created to the time it have been
SNAPSHOT
destroyed.
Tracks the total time a load balancer policy has been created to the time it has been
LOAD_BALANCER_POLICY
removed. Cloud.com does not track whether a VM has been assigned to a policy.
Tracks the time from when a port forwarding rule was created until the time it was
PORT_FORWARDING_RULE
removed.
NETWORK_OFFERING
The time from when a network offering was assigned to a VM until it is removed.
VPN_USERS
The time from when a VPN user is created until it is removed.
8
Usage Records
Currently there is no UI which displays usage records. API call listUsageRecords fetches
usage records. Usage Records show the amount of resources consumed by guest
instances.
•
•
•
•
•
•
•
•
Allocated & Running VMs
Network Usage
IP Address
Disk Volume
Template, ISO and Snapshot
Load Balancer or Port Forwarding
Network Offering
VPN User
Footer – Apply across document
9
Vm Usage Record Format
For running and allocated virtual machine usage, the following fields exist in a usage
record:
account –
name of the account
accountid –
ID of the account
domainid –
ID of the domain in which this account resides
zoneid –
Zone where the usage occurred
description –
A string describing what the usage record is tracking
usage –
String representation of the usage, including the units of usage (e.g. 'Hrs' for VM running time)
usagetype –
A number representing the usage type (see Usage Types)
rawusage –
A number representing the actual usage in hours
virtualMachineId – The ID of the virtual machine
name –
The name of the virtual machine
offeringid –
The ID of the service offering
templateid –
The ID of the template or the ID of the parent template. The parent template value is present when the
current template was created from a volume.
usageid –
Virtual machine
type –
Hypervisor
startdate, enddate – The range of time for which the usage is aggregated; see Dates in the Usage Record
Footer – Apply across document
10
Usage Record Example
Allocated VM
Every time a VM is created VM.CREATE event is logged. Similarly VM.DESTROY event is logged when VM is
destroyed. Corresponding entries in cloud.usage_event table would like the following:
+———————-——————-——-———–————-———–———–—-———-+
| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |
+———————-——————-——-———–————-———–———–—-———-+
|
11
|
VM.CREATE
|
2
|
2011-02-09
11:15:27
|
1
|
5
|
i-2-5-KK
|
7
|
2
|
NULL
|
0
|
|
12
|
VM.DESTROY
|
2
|
2011-03-09
10:16:18
|
1
|
5
|
i-2-5-KK
|
7
|
2
|
NULL
|
0
|
+———————-——————-——-———–————-———–———–—-———-+
Duration between VM.CREATE and VM.DESTROY is reported as Allocated VM usage.
Running VM
Usage for Running VM is implemented similar to allocated VM usage. Events VM.start and VM.stop are
tracked here instead.
+———–———-——————-——-———–————-———–———–—-———-+
| id | type
| account_id | created
| zone_id | resource_id | resource_name | offering_id | template_id | size | processed |
+———–———-——————-——-———–————-———–———–—-———-+
| 12 | VM.START |
2 | 2011-02-09 11:16:18 |
1 |
5 | i-2-5-KK
|
7 |
2 | NULL |
0 |
| 15 | VM.STOP |
2 | 2011-02-09 12:00:24 |
1 |
5 | i-2-5-KK
|
7 |
2 | NULL |
0 |
+———–———-——————-——-———–————-———–———–—-———-+
Duration between VM.START and VM.STOP is reported as Running VM usage.
Footer – Apply across document
11
How we can access usage data
Footer – Apply across document
12
CloudStack API
•
The usage records can be accessed through the CloudStack API .
•
This is best done using the authentication port 8080 and requires both the API Key
and Secret Key and for the API request to be signed
•
CloudStack provides the following Root Admin API calls and responds with XML
and JSON responses:
1. listUsageTypes – lists available Usage Types
2. listUsageRecords - provides Usage records for a date range
3. generateUsageRecords - asynchronous usage record generation
Footer – Apply across document
13
ListUsageRecords API
The base API request is:
http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y
yyy-MM-dd&enddate=yyyy-MM-dd&signature
• Optional request parameters:
Footer – Apply across document
14
Example response from listUsageRecords
<listusagerecordsresponse>
<count>1816</count>
<usagerecord>
<account>user5</account>
<accountid>10004</accountid>
<domainid>1</domainid>
<zoneid>1</zoneid>
<description>i-3-4-WC running time (ServiceOffering: 1) (Template: 3)</description>
<usage>2.95288 Hrs</usage>
<usagetype>1</usagetype>
<rawusage>2.95288</rawusage>
<virtualmachineid>4</virtualmachineid>
<name>i-3-4-WC</name>
<offeringid>1</offeringid>
<templateid>3</templateid>
<usageid>245554</usageid>
<type>XenServer</type>
<startdate>2009-09-15T00:00:00-0700</startdate>
<enddate>2009-09-18T16:14:26-0700</enddate>
</usagerecord>
</listusagerecordsresponse>
Footer – Apply across document
15
CloudMonkey CLI
•
CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell
or from within a shell script to call the API.
•
It can output the Usage Data in both JSON and Tabular form to a file or pipe it to
another application for further processing.
•
Filtering can be used to limit the result set.
•
It supports argument passing and shell automation.
•
The host, port, apikey and secretkey can be configured on setup.
Footer – Apply across document
16
CloudMonkey list usagerecords
The command to retrieve Usage Records is:
cloudmonkey list usagerecords domainid=7d8707bf-e6ad-11e3-a7e8-f245a5b3ba0f startdate=2014-08-10
enddate=2014-08-11 accountid=efda79de-ff01-4de2-bbe9-56b0ec366a07
Footer – Apply across document
17
SQL
•
The usage records can be also accessed by directly querying the
cloud_usage.cloud_usage MySQL table in CloudStack
•
Usage Data can be exported in a CSV format from a SELECT query
•
The SQL query can become complex if the ID fields in the data need to be dereferenced
•
Typically multiple passes of the query results may be required to extract the
required information
Footer – Apply across document
18
CloudStack Message Queue
•
CloudStack 4.1 started publishing events onto a message queue Uses RabbitMQ
as the message broker, but likely to work with other AMQP- based brokers .
•
Use case: A third-party cloud usage solution can subscribe to CloudStack
events and generate usage data which can be consumed by their billing software
.
For more details Please refer this link
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/EventBus
Footer – Apply across document
19
Customer Use Cases
Footer – Apply across document
20
MS Excel
MS Excel
 MS Excel can be used to analyse CloudStack Usage Records
 The usage records are imported from the cloud_usage table into MS Excel
via an MySQL ODBC connection
 Either PivotTables or reports can be used to present the data
 This is convenient and great for integration testing between CloudStack and
external systems
Footer – Apply across document
21
Splunk
•
Splunk is an operational intelligence tool that was initially used for collecting
and indexing infrastructure logs
•
We then connected splunk to the CloudStack Usage database and automated
the collection and indexing of Usage Data
•
Setup feature-rich dashboards and the capability to drill down into the Usage
Data for BI reporting
Footer – Apply across document
22
Amysta
Provides usage visibility, cost control and IT billing for Private and Hybrid Clouds Dashboard
Integrated in the CloudStack UI Pricing, Alerting, Reporting, Capacity Mgmt Amysta
consolidates public cloud expenses e.g. AWS and private cloud consumption in one
dashboard view
Fore more details please refer this link:
https://www.youtube.com/watch?v=ZA-VPb7WAy8
http://www.slideshare.net/buildacloud/apaliaamysta-cloud-usage-metering-and-billing
Footer – Apply across document
23
SunGard Cloud Portal
Footer – Apply across document
24
Ubersmith's
Ubersmith's CloudStack Billing Service Module allows you to have Ubersmith bill for
CloudStack usage.
Fore more details please refer this link :
http://www.ubersmith.com/kbase/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=161&nav=0
Footer – Apply across document
25
Thank You!