Charts on dashboard in Manageiq Capablanca : scheme of generating content and fixing problem

20:54 4 Comments

 A lot of people, which use Manageiq Capablanca, faced with problem of charts on dashboard. In this post i will descibe you scheme of building charts and solution of problem with dashboard. Let's go!


Scheme of building content of charts



In generating of contents of charts all functions of model MiqWidget  are involved.
But the most interesting:
-   generate_content
- generate_report
- generate_report_result
In function generate_content, firstly function generate_report is calling and then generate_report_result is calling. So, we have content to build our charts. Then we go to MiqWidget::ChartContent
--
class MiqWidget::ChartContent < MiqWidget::ContentGeneration
  def generate(user_or_group)
    theme = user_or_group.settings.fetch_path(:display, :reporttheme) if user_or_group.kind_of?(User)

    # TODO: default reporttheme to MIQ since it doesn't look like we ever change it
    theme ||= "MIQ"
    report.to_chart(theme, false, MiqReport.graph_options)
    report.chart
  end
end

We can see, that all charts use standard graphic options(size, colours and other).
So, then we should to know more about function to_chart:

--
module MiqReport::Formatters::Graph
  extend ActiveSupport::Concern

  module ClassMethods
  ...
  def to_chart(theme = nil, show_title = false, graph_options = nil)
    ReportFormatter::ReportRenderer.render(Charting.format) do |e|
      e.options.mri           = self
      e.options.show_title    = show_title
      e.options.graph_options = graph_options unless graph_options.nil?
      e.options.theme         = theme
    end

  end
end
Charting.format = jqplot ( for all charts). But what about ReportFormatter::ReportRenderer.render?

--
module ReportFormatter
  class ReportRenderer < Ruport::Controller
    stage :document_header, :document_body, :document_footer
    finalize :document
    options { |o| o.mri = o.show_title = o.theme = o.table_width = o.alignment = o.graph_options = nil }
  end
end

How we can see, ReportFormatter::ReportRenderer use standard library ruport, which use module ChartCommon. And this module, finally, build contents of charts. Look at module ChartCommon and some important functions of it:
-
    def build_document_body
      return no_records_found_chart if mri.table.nil? || mri.table.data.blank?

      # find the highest chart value and set the units accordingly for large disk values (identified by GB in units)
      maxcols = 8
      divider = 1
      if graph_options[:units] == "GB" && !graph_options[:composite]
        maxval = 0
        mri.graph[:columns].each_with_index do |col, col_idx|
          next if col_idx >= maxcols
          newmax = mri.table.data.collect { |r| r[col].nil? ? 0 : r[col] }.sort.last
          maxval = newmax if newmax > maxval
        end
        if maxval > 10.gigabytes
          divider = 1.gigabyte
        elsif maxval > 10.megabytes
          graph_options[:units] = "MB"
          divider = 1.megabyte
        elsif maxval > 10.kilobytes
          graph_options[:units] = "KB"
          divider = 1.kilobyte
        else
          graph_options[:units] = "Bytes"
          graph_options[:decimals] = 0
          divider = 1
        end
        mri.title += " (#{graph_options[:units]})" unless graph_options[:units].blank?
      endfun = case graph_options[:chart_type]
            when :performance then :build_performance_chart # performance chart (time based)
            when :util_ts     then :build_util_ts_chart     # utilization timestamp chart (grouped columns)
            when :planning    then :build_planning_chart    # trend based planning chart
            else                                            # reporting charts
              mri.graph[:mode] == 'values' ? :build_reporting_chart_numeric : :build_reporting_chart
            end
      method(fun).call(maxcols, divider)
    end
    
    def build_reporting_chart_other
      save_key   = nil
      counter    = 0
      categories = []                      # Store categories and series counts in an array of arrays
      mri.table.data.each_with_index do |r, d_idx|
        if d_idx > 0 && save_key != r[mri.sortby[0]]
          save_key = nonblank_or_default(save_key)
          categories.push([save_key, counter])    # Push current category and count onto the array
          counter = 0
        end
        save_key = r[mri.sortby[0]]
        counter += 1
      end
      # add the last key/value to the categories and series arrays
      save_key = nonblank_or_default(save_key)
      categories.push([save_key, counter])        # Push last category and count onto the array

      categories.sort! { |a, b| b.last <=> a.last }
      (keep, show_other) = keep_and_show_other
      if keep < categories.length                      # keep the cathegories w/ highest counts
        other = categories.slice!(keep..-1)
        ocount = other.reduce(0) { |a, e| a + e.last } # sum up and add the other counts
        categories.push(["Other", ocount]) if show_other
      end

      series = categories.each_with_object(
        series_class.new(pie_type? ? :pie : :flat)) do |cat, a|
        a.push(:value => cat.last, :tooltip => "#{cat.first}: #{cat.last}")
      end

      # Pie charts put categories in legend, else in axis labels
      limit = pie_type? ? LEGEND_LENGTH : LABEL_LENGTH
      categories.collect! { |c| slice_legend(c[0], limit) }
      add_axis_category_text(categories)
      add_series(mri.headers[0], series)
    end 

Functions of module ChartCommon form content of charts belonging to MiqWidget, and this content is saved in database in table miq_widgets_contents.
Then, when we want to see at charts on dashboard this content is parsed by YAMl parser and converted in charts. Look at page of charts:

-
-#
  Parameters:
    widget -- MiqWidget instance
- width  ||= 350
- height ||= 250
.panel-body.chart_widget{:style => "text-align: center; padding: 0"}
  .mc{:id => "dd_w#{widget.id}_box",
    :style => @sb[:dashboards][@sb[:active_db]][:minimized].include?(widget.id) ? 'display:none' : ''}

    - if widget.contents_for_user(current_user).contents.blank?
      = _('No chart data found')
      \. . .
    - datum = widget.contents_for_user(current_user).contents
    - if Charting.data_ok?(datum)
      -# we need to count all charts to be able to display multiple
      -# charts on a dashboard screen

      - datum = widget.contents_for_user(current_user).contents
      - WidgetPresenter.chart_data.push(:xml => datum)
      - chart_index = WidgetPresenter.chart_data.length - 1
      - chart_data = YAML.load(datum) if Charting.backend == :jqplot

      = chart_local(chart_data,
                    :id     => "miq_widgetchart_#{chart_index}".html_safe,
                    :width  => width,
                    :height => height)
    - else
      = _('Invalid chart data. Try regenerating the widgets.')
 

Problem with charts and its solution

When you install new version of Manageiq - Capablanca, you can see this on dashboard:


The reason for this problem is that in ChartCommon data is generated
in the form of Hash, but we need YAML.
So, you should add one correction in MiqWidget::ChartContent:

-
class MiqWidget::ChartContent < MiqWidget::ContentGeneration
  def generate(user_or_group)
    theme = user_or_group.settings.fetch_path(:display, :reporttheme) if user_or_group.kind_of?(User)

    # TODO: default reporttheme to MIQ since it doesn't look like we ever change it
    theme ||= "MIQ"
    report.to_chart(theme, false, MiqReport.graph_options)
    report.chart.to_yaml
  end
end

 
Result:


4 comments:

CFME Reports: scheme and extensions

14:20 0 Comments

About scheme of Reports


It's necessary to say in this topic about model MiqExpression, which is stored information about tables, which are used for Reports. This model have four enumerators related with Reports:

1. Enumerator @@base_tables(not fully):
---
@@base_tables = %w{
    Network
    AuditEvent
    AvailabilityZone
    BottleneckEvent
    Chargeback
    CloudResourceQuota
    CloudTenant
    Compliance
    ConfiguredSystemForeman
    ConfigurationManager
    EmsCloud
    EmsCluster
    EmsClusterPerformance
    EmsEvent
    EmsInfra
    ExtManagementSystem
    Flavor
    Host
    Vm
    ...

This enumerator includes names of models, on which reports are built. We can see at snapshot:

2. Enumerator @@include_tables (not fully too):
---
@@include_tables = %w{
    advanced_settings
    audit_events
    availability_zones
    cloud_networks
    cloud_resource_quotas
    cloud_tenants
    compliances
    compliance_details
    configuration_profiles
    configuration_managers
    configured_systems
    customization_scripts
    customization_script_media
    customization_script_ptables
    disks
    networks
    hardware
    ...

This enumerator includes tables, which are not base for reports, but we can see and use these tables, because they are part of @@base_tables. For example, table networks belongs to table hardware, and this table belongs to table of model Vm. So, networks and hardware are included in @@include_tables, and we can see them in available tables for reports. See on snapshot:

 3. Enumerator EXCLUDE_COLUMNS (not fully too):
---
EXCLUDE_COLUMNS = %w{
    ^.*_id$
    ^id$
    ^min_derived_storage.*$
    ^max_derived_storage.*$
    assoc_ids
    capture_interval
    filters
    icon

    intervals_in_rollup

    max_cpu_ready_delta_summation
    max_cpu_system_delta_summation
    max_cpu_used_delta_summation
    max_cpu_wait_delta_summation
    max_derived_cpu_available
    max_derived_cpu_reserved
    max_derived_memory_available
    max_derived_memory_reserved

    memory_usage
    ...

This enumerator includes fields, which should not be displayed in reports. For example, we can't see field password in report, which is based on model EVM User,but  password stores in table of this model.

4. Enumerator EXCLUDE_EXCEPTIONS:
---
EXCLUDE_EXCEPTIONS = %w{
    capacity_profile_1_memory_per_vm_with_min_max
    capacity_profile_1_vcpu_per_vm_with_min_max
    capacity_profile_2_memory_per_vm_with_min_max
    capacity_profile_2_vcpu_per_vm_with_min_max
    chain_id
    guid
  }
 
This enumerator includes fields, which should not be displayed in reports too.

Our extension for Reports


Now, we know scheme of Reports and we can add custom extension.For example:
Our task: We can click on VM in report, which includes VM Name and IP adress, and we can go to this VM. Finally, IP adress should be displayed in base model for reports Perfomance - VMs.
Our actions:

---
class VmPerformance < MetricRollup
  default_scope { where "resource_type = 'VmOrTemplate' and resource_id IS NOT NULL" }

  belongs_to :host,        :foreign_key => :parent_host_id
  belongs_to :ems_cluster, :foreign_key => :parent_ems_cluster_id
  belongs_to :storage,     :foreign_key => :parent_storage_id
  belongs_to :vm,          :foreign_key => :resource_id, :class_name => 'VmOrTemplate'

  virtual_column :ipaddresses,    :type => :string_set

  def ipaddresses
    return self.vm.hardware.ipaddresses
  end

end
- new virtual column ipaddresses with type string_set;
 - definition of ipaddresses (we use vm - model VmPerformance belongs to model of this table).

 Results:

0 comments:

SmartState analysis scheme, restrictions, extensions

16:50 1 Comments

Scheme of Smart State Analysis
The intent of Smart State Analysis is to collect information from Virtual machine and save it in models.
These Models are:
 - Nerwork
 - OperatingSystem
 - Account
 - SystemService
 - GuestApplication
 - Patch
Smart State Analysis system generates a few xml-files with information from VM. Then data is saved using the function add_elements.
And these xml-files are saved in /var/www/miq/vmdb/data/metadata/*.( Also, these xml-files are stored one day and if VM is scanned several times per day, file will be replaced).

It is necessary to say about modules,which collects information and group into following categories.
There are five categories:
 - system
 - services
 - accounts
 - software
 - ntevents
 - vmconfig

Used module depends on OS:

Collection category Source module (Windows) Source module (Linux)
systemMiqWin32::SystemMiqLinux::System
services MiqWin32::Services MiqLinux::InitProcs
accounts MiqWin32::Accounts MiqLinux::Users
software MiqWin32::Software MiqLinux::Packages
ntevents Win32EventLog -
vmconfig MIQExtract MIQExtract

Some restrictions of Smart State Analysis

After exploring Smart State Analysis, restrictions were found, and these restrictions are not documented. More about them:
  1. System of Smart State Analysis supports scanning of files with network configuration in OS: Redhat, Ubuntu and Fedora and emulator Herkules.
  2. System of Smart State Analysis checks only network interface(ifcfg-eth0).
  3. If in configuration file parameters are with '',  System of Smart State Analysis won't parse it correctly.
Our extensions to these restrictions of Smart State Analysis

 The solution of required tasks is creation of extensions of standart functions. Look at new custom functions, which are based on function networking_redhat:
 - networking_custom
 - dhcp_configuration

 
---
SUSEIFCFGFILE = "/etc/sysconfig/network/"
SUSEDHCLIENTFILE = "/var/lib/dhcpcd/"
REDHATIFCFGFILE = "/etc/sysconfig/network-scripts/"
REDHATDHCLIENTFILE = "/var/lib/dhclient/"

def self.dhcp_configuration(ifcfg, attrs, fs)
  dhcpcfg = "dhcpcd-eth" + ifcfg.delete("ifcfgeth-") +".info"
  self.read_file(fs,File.join(SUSEDHCLIENTFILE, dhcpcfg)) do |line|
    case line
      when  /^\s*IPADDR\s*=\s*(.*)$/        then attrs[:ipaddress] = $1
      when  /^\s*NETMASK\s*=\s*(.*)$/       then attrs[:subnet_mask] = $1
      when  /^\s*BROADCAST\s*=\s*(.*)$/     then attrs[:broadcast] = $1
      when  /^\s*NETWORK\s*=\s*(.*)$/       then attrs[:network] = $1
      when  /^\s*GATEWAYS\s*=\s*(.*)$/       then attrs[:default_gateway] = $1
      when  /^\s*DNSDOMAIN\s*=\s*(.*)$/       then attrs[:domain] = $1
      when  /^\s*DNSSERVERS\s*=\s*(.*)$/       then attrs[:dns_server] = $1
      when  /^\s*DHCPSIADDR\s*=\s*(.*)$/       then attrs[:dhcp_server] = $1
    end
  end
  if (attrs.has_key?(:dhcp_server)==false)
    self.read_file(fs,File.join(REDHATDHCLIENTFILE, "dhclient.leases")) do |line|
      case line
        when /^\s*fixed-address\s*(.*)\;$/                 then attrs[:ipaddress] = $1
        when /^\s*option subnet-mask\s*(.*)\;$/            then attrs[:subnet_mask] = $1
        when /^\s*option routers\s*(.*)\;$/                then attrs[:default_gateway] = $1
        when /^\s*option domain-name-servers\s*(.*)\;$/    then attrs[:dns_server] = $1
        when /^\s*option dhcp-server-identifier\s*(.*)\;$/ then attrs[:dhcp_server] = $1
        when /^\s*option domain-name\s*"*(.*)"\;$/         then attrs[:domain] = $1
        when /^\s*expire\s*[0-9]?\s*(.*)\;$/               then attrs[:lease_expires] = $1
        when /^\s*renew\s*[0-9]?\s*(.*)\;$/                then attrs[:lease_obtained] = $1
      end
    end
  end

  attrs
end


def self.networking_custom(fs,attrs,os)
  ifcfg = "ifcfg-eth"
  if (os.eql?("suse"))
    ifcfgfile = SUSEIFCFGFILE
  elsif (os.eql?("redhat") || os.eql?("fedora"))
    ifcfgfile = REDHATIFCFGFILE
  end
  for i in 0..20
    ifcfg_i = ifcfg + i.to_s
    self.read_file(fs,File.join(ifcfgfile, ifcfg_i)) do |line|
      case line
        when  /^\s*BOOTPROTO='?dhcp'?\s*(.*)$/    then attrs[:dhcp_enabled] = 1
        when  /^\s*BOOTPROTO='?static'?\s*(.*)$/  then attrs[:dhcp_enabled] = 0
        when  /^\s*DEVICE\s*=\s*(.*)$/        then attrs[:device] = $1
        when  /^\s*HWADDR\s*=\s*(.*)$/        then attrs[:hwaddr] = $1
        when  /^\s*IPADDR\s*=\s*(.*)$/        then attrs[:ipaddress] = $1
        when  /^\s*NETMASK\s*=\s*(.*)$/       then attrs[:subnet_mask] = $1
        when  /^\s*BROADCAST\s*=\s*(.*)$/     then attrs[:broadcast] = $1
        when  /^\s*NETWORK\s*=\s*(.*)$/       then attrs[:network] = $1
      end
    end
    if (attrs.has_key?(:dhcp_enabled)==true) then
      break
    end
  end
  if (attrs[:dhcp_enabled] == 1)
    attrs = self.dhcp_configuration(ifcfg, attrs,fs)
  end
  self.fix_attr_values(attrs)
  attrs
end

- Function self.networking_custom finds first network interface with BOOTPROTO value (first configured interface)
- If BOOTPROTO=dhcp, then discover DHCP configuration with dhcp_configuration function, currently dhcpd and dhclient supported.
 

1 comments:

CloudForms API extension: custom model

16:54 0 Comments

Today we study new way to extend CloudForms API. Using this method, we will create new model, which will collect information from other models and transform it as we need. There is not easy way, but it's worth to try.
For example, we will create two models - MonthBilling and AllBilling.
Firstly, we look at model MonthBilling. This model will collect of information for the last month. MonthBilling includes: name of services, number of vms, and cpu usage, memory usage, net usage, disk usage and total cost. This model is based on model MetricRollup and it contains information about accounts for cpu, memory, usage internet, usage disk space. Also, MetricRollup stores information about resources, for which the accounts are and time, during which the accounts are calculated.
 In interface of MonthBilling we will use ActsAsArModel which is provide dynamic AR-like model, where columns defined by hashes.

---
# Model MonthBilling
class MonthBilling < ActsAsArModel

@us = nil

  set_columns_hash(
    :id              => :integer,
    :name            => :string,
    :vms             => :integer,
    :cpu             => :float,
    :mem             => :float,
    :disk            => :float,
    :net             => :float,
    :tco             => :float
  )

  def self.sortable?
    true
  end

  def self.find(*args)
    res = []
    arg = *args
    method = *args
    first_arg = args.first.to_s
    if (first_arg.eql?("all"))
      if (args[1].nil?)
        id_user = User.find_by_userid(@us)
        services = Service.where(evm_owner_id: id_user.id)
        res = ApiController.helpers.add_metrics(services,"month")
      else
        size_conditions = args[1][:conditions].size
        res_string = ""
        for i in 9 .. size_conditions
          if ((args[1][:conditions][i].to_i)== 0 && !(args[1][:conditions][i].eql?("0")))
            break
          end
          res_string+=args[1][:conditions][i]
         end
        if (args[1][:conditions][i].eql?(","))
          res = self.find(:all)
        else
          res_number = res_string.to_i
          res = self.find(res_number)
        end
      end
    else
    id_user = User.find_by_userid(@us)
      services = Service.where(id: args.first.to_i, evm_owner_id: id_user.id)
      res = ApiController.helpers.add_metrics(services,"month")
    end
    res
  end

  def self.auth_initiliaze(user)
  @us = user
  end

  def self.count(*args)
    id_user = User.find_by_userid(@us)
    Service.where(evm_owner_id: id_user.id).count
  end


  def self.scoped(options = {})
    self.find(:all)
  end

end
You have to implement following methods:
 -self.count(*args): this method is needed for quick information about count of billing. Billings are counted for each service, and count of billing is equal to the count of services.
 - self.auth_initiliaze(user): this method is necessary for searching services, which are belong to this user.@us is initialized before, in renderer.rb
- self.find(*args): in this function search is based on arguments. If first  argument - id, then then result is based on this id (service is searched by id). If first  argument - :all, and second argument is null, then result is based on all services and all information about. If first  argument - :all, and second argument is not null, then second argument is parsed, and then result is based on results of parsing(if it is id in second arguments, then result is based on this id. In another case, result is based on all services). Required by ActsAsArModel.
- self.scoped(options = nil): required by RBAC (${VMDB}/app/models/rbac.rb)
- attribute id: it's required to create correct "href"s.
In MonthBilling we will show billings for services belonging to user. So, to know what user work with our API, we will add function self.auth_initiliaze in  ${VMDB}/app/helpers/api_helper/renderer.rb
---
# $VMDB/app/helpers/api_helper/renderer.rb
module ApiHelper
  module Renderer
    #
    # Helper proc for rendering a collection of type specified.
    #
    def render_collection_type(type, id, is_subcollection = false)
      klass = collection_class(type)
      if (klass.respond_to?(:auth_initiliaze))
      klass.auth_initiliaze(@auth_user)
      end
.....
Model MonthBilling uses important function - add_metrics. We save this method in ${VMDB}/app/helpers/api_helper/custom_helper.rb
---
# $VMDB/app/helpers/api_helper/custom_helper.rb
def add_metrics(services,time)
      $api_log.info("[DBG] start IBAHelper.add_metrics()")
      res = []
      required_time = nil
      if (time.to_s.eql?("month"))
        required_time = Time.now - 1.month
      elsif (time.to_s.eql?("week"))
        required_time = Time.now - 1.week
      elsif (time.to_s.eql?("day"))
        required_time = Time.now - 1.day
      end
      array_vms = []
      hash_services = {}
      hash_billings = {}
      hash_billings = Hash.new{|h,k| h[k]=Hash.new(&h.default_proc)}
      services.map{|service|
        hash_billings[service][:cpu] = 0
        hash_billings[service][:mem] = 0
        hash_billings[service][:net] = 0
        hash_billings[service][:disk] = 0
        hash_billings[service][:tco] = 0
        service.vms.map{|vm| array_vms.push(vm.id) }
        service.vms.map{|vm| hash_services[vm.id] = service }
      }
      resources = MetricRollup.select("resource_id, SUM(cpu_usage_rate_average) as cpu_usage_rate_average, SUM(mem_usage_absolute_average)as mem_usage_absolute_average, SUM(disk_usage_rate_average) as disk_usage_rate_average,SUM(net_usage_rate_average) as net_usage_rate_average").where(resource_type: 'VmOrTemplate', capture_interval_name: 'hourly', timestamp: (required_time..Time.now), resource_id: array_vms).group(:resource_id)
      hash_services.each_pair do |key, value|
        resource = resources.where(resource_id: key.to_i)
        resource.each do |element|
          $api_log.info("in element - #{key} => #{value}")
          hash_billings[value][:cpu] += element.cpu_usage_rate_average.to_f
          hash_billings[value][:mem] += element.mem_usage_absolute_average.to_f
          hash_billings[value][:net] += element.net_usage_rate_average.to_f
          hash_billings[value][:disk]+= element.disk_usage_rate_average.to_f
          hash_billings[value][:tco] += element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
        end
      end
      hash_billings.each_pair do |key, value|
        if (time.to_s.eql?("month"))
          $api_log.info("if (time.to_s.eql?(month)) - billinsg")
          res.push(MonthBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        elsif (time.to_s.eql?("week"))
          $api_log.info("if (time.to_s.eql?(week) - billings")
          res.push(WeekBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        elsif (time.to_s.eql?("day"))
          $api_log.info("if (time.to_s.eql?(day)) = billings")
          res.push(DayBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        end
      end
    $api_log.info("[DBG] end IBAHelper.add_metrics()")
    res
    end
1. Firstly, time is calculated, according to argument time.
2.Then hash of billings is initilizated. Hash contains information about id, name of services and count of vms. Other attributes are zero. Also, information about necessary id of vms is collected.
3. Information about cpu, memory and etc. is stored from model MetricRollup, where id = id of vm, capture_interval - hourly, and information is collected for the last month.In hash of billings information about cpu, memory and etc. is summed.
4. According to argument time, new MonthBilling is created. As you can see, according to argument time, you can create other model like this, for example WeekBilling. And, you should to desribe model in ${VMDB}/config/api.yml
---
# $VMDB/config/api.yml
:billings_monthly:
    :description: Billings
    :options:
    - :collection
    :methods: *70174834086080
    :klass: MonthBilling
Then we will restart CFME:

$> cd /var/www/miq/vmdb/
$> rake evm:restart
And now you can see result.
Go to the page: {hostname}/api/billings_monthly
Output:
{"name":"billings_monthly","count":5,"subcount":5,"resources":[{"href":"https://{hostname}/api/billings_monthly/12"},{"href":"https://{hostname}/api/billings_monthly/13"},{"href":"https://{hostname}/api/billings_monthly/8"},{"href":"https://{hostname}/api/billings_monthly/11"},{"href":"https://{hostname}/api/billings_monthly/10"}]}


Let's go back to AllBilling. This model is based on collection of information for the last month per hour. This model is subcollection, and information in this model is based on one service. This model includes: name of service, number of vms, and cpu usage, memory usage, net usage, disk usage and total cost.

---
# $VMDB/app/models/all_billing.rb

class AllBilling < ActsAsArModel

@us = nil

  set_columns_hash(
    :id              => :integer,
    :name            => :string,
    :vms             => :integer,
    :cpu             => :float,
    :mem             => :float,
    :disk            => :float,
    :net             => :float,
    :tco             => :float
  )

  def self.find(*args)
    arg = *args
    method = *args
    first_arg = args.first.to_s
    res = []
    if (first_arg.eql?("all"))
      index = 0
      if (args[1].nil?)
        id_user = User.find_by_userid(@us)
        services = Service.where(evm_owner_id: id_user.id)
        res = ApiController.helpers.add_billings(services)
      else
        size_conditions = args[1][:conditions].size
        res_string = ""
        for i in 9 .. size_conditions
          if ((args[1][:conditions][i].to_i)== 0 && !(args[1][:conditions][i].eql?("0")))
            break
          end
          res_string+=args[1][:conditions][i]
        end
        if (args[1][:conditions][i].eql?(","))
          res = self.find(:all)
        else
          res_number = res_string.to_i
          res = self.find(res_number)
        end
      end
    else
      first_arg = args.first
      res = ApiController.helpers.add_billings_by_id(args.first)
    end
    res
  end


  def self.auth_initiliaze(user)
  @us = user
  end

  def self.count(*args)
    $api_log.info("[DBG] start self.count()")
    0
  end


  def self.scoped(options = {})
   $api_log.info("[DEBUG] (Billings) self.scoped("+options.inspect+")")
   self.find(:all)
  end

end 
 
As this is subcollection, we need to add module Billing in file billings.rb in ${VMDB}/app/controllers/api_controller. When we want to see subcollection, results are created by function {name_module}__query_resource
---
# $VMDB/app/controllers/api_controllers/billings.rb
class ApiController
  module Billings
    #
    # Rates Subcollection Supporting Methods
    #
    def billings_query_resource(data)
      $api_log.info("billings_query_resource")
      data ? ApiController.helpers.add_billings_all(data.id) : {}
    end
  end
end

And we will include this module in ApiController(${VMDB}/app/controller/api_controller.rb)
---
# $VMDB/app/controllers/api_controller.rb
include_concern 'Billings'
Almost all methods are described before. And what about add_billings and add_billings_by_id and add_billings_all. These methods are in ${VMDB}/app/helpers/api_helper/custom_helper.rb
---
# $VMDB/app/helpers/api_helper/custom_helper.rb
 
 def add_billings_all(add_id)
    services = Service.where(id: add_id)
    res = []
    array_vms = []
    hash_billings = {}
    hash_services = {}
    $api_log.info("#{services}")
    services.map{|service|
      service.vms.map{|vm| array_vms.push(vm.id) }
      service.vms.map{|vm| hash_services[vm.id] = service }
      }
     resources = MetricRollup.select("id, resource_id, cpu_usage_rate_average, mem_usage_absolute_average,disk_usage_rate_average,net_usage_rate_average").where(resource_id: array_vms, capture_interval_name: 'hourly').group(:id, :resource_id)
     resources.each do |element|
       hash_billings[:id] =  element.id
       hash_billings[:name] = (hash_services.fetch(element.resource_id)).name
       hash_billings[:vms] = (hash_services.fetch(element.resource_id)).vms.size
       hash_billings[:cpu] = element.cpu_usage_rate_average.to_f
       hash_billings[:mem] = element.mem_usage_absolute_average.to_f
       hash_billings[:net] = element.net_usage_rate_average.to_f
       hash_billings[:disk]= element.disk_usage_rate_average.to_f
       hash_billings[:tco] = element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
       res.push(AllBilling.new(id: hash_billings[:id], name: hash_billings[:name], vms: hash_billings[:vms], cpu: hash_billings[:cpu], mem: hash_billings[:mem], disk: hash_billings[:disk], net: hash_billings[:net], tco: hash_billings[:tco]))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
   end

def add_billings(services)
    res = []
   array_vms = []
   hash_services = {}
   hash_billings = {}
   hash_billings = Hash.new{|h,k| h[k]=Hash.new(&h.default_proc)}
   services.map{|service|
     service.vms.map{|vm| array_vms.push(vm.id) }
     service.vms.map{|vm| hash_services[vm.id] = service }
   }
   resources = MetricRollup.select("id, resource_id,  cpu_usage_rate_average, mem_usage_absolute_average, disk_usage_rate_average, net_usage_rate_average").where(resource_type: 'VmOrTemplate', capture_interval_name: 'hourly', timestamp: ((Time.now - 1.month)..Time.now), resource_id: array_vms).group(:id, :resource_id)
     resources.each do |element|
       hash_billings[:id] =  element.id
       hash_billings[:name] = (hash_services.fetch(element.resource_id)).name
       hash_billings[:vms] = (hash_services.fetch(element.resource_id)).vms.size
       hash_billings[:cpu] = element.cpu_usage_rate_average.to_f
       hash_billings[:mem] = element.mem_usage_absolute_average.to_f
       hash_billings[:net] = element.net_usage_rate_average.to_f
       hash_billings[:disk]= element.disk_usage_rate_average.to_f
       hash_billings[:tco] = element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
       res.push(AllBilling.new(id: hash_billings[:id], name: hash_billings[:name], vms: hash_billings[:vms], cpu: hash_billings[:cpu], mem: hash_billings[:mem], disk: hash_billings[:disk], net: hash_billings[:net], tco: hash_billings[:tco]))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
end

  def add_billings_by_id(add_id)
    res = []
    serv = nil
    services = Service.find(:all)
    resources = MetricRollup.select("id, resource_id, cpu_usage_rate_average, mem_usage_absolute_average,disk_usage_rate_average,net_usage_rate_average").where(id: add_id).group(:id, :resource_id)
    resources.each do |element|
      services.each do |service|
        service.vms.each do |vm|
          if (vm.id == element.resource_id)
          serv = service
          end
        end
      end
     res.push(AllBilling.new(id: element.id, name: serv.name, vms: serv.vms.size, cpu: element.cpu_usage_rate_average.to_f, mem: element.mem_usage_absolute_average.to_f, disk: element.disk_usage_rate_average.to_f, net: element.net_usage_rate_average.to_f, tco: (element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.disk_usage_rate_average.to_f + element.net_usage_rate_average.to_f)))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
  end

Methods add_billings_all and add_billings are based on the same things:
1. Information about services and id of vms is collected in hash_services and array_vms.
2. Information collected from MetricRollup by id of vms.
3. All information about billings are in hash_billings.
4. New AllBilling is based on information of hash_billings.
In method add_billings_by_id we have id of element from MetricRollup and we will find needed element and service.
Then, we should describe this subcollection in ${VMDB}/config/api.yml
---
# $VMDB/config/api.yml

:billings:
    :description: Billings
    :options:
    - :subcollection
    :methods: *70174834086080
    :klass: AllBilling
 
:services:
    :description: Services
    :options:
    - :collection
    :methods: *70174834084700
    :klass: Service
    :subcollections:
    - :billings
    - :tags
Restart CFME:

$> cd /var/www/miq/vmdb/
$> rake evm:restart
It's time to see results.
Go to the page: https://{hostname}.128/api/services/8/billings/
Output:
{"name":"billings","count":0,"subcount":128,"resources":[{"href":"https://{hostname}/api/services/8/billings/368"},{"href":"https://{hostname}/api/services/8/billings/780"}]}
Go to the page: https://{hostname}.128/api/services/8/billings/368
Output:
{"href":"https://{hostname}/api/services/8/billings/368","id":368,"name":"AnnKurnik-20150918-070008","vms":1,"cpu":1.39155555555556,
"mem":0.871333333333333,"disk":0.0,
"net":10.1222222222222,"tco":12.385111111111094}

0 comments:

CloudForms API extension: fast way

17:41 0 Comments

Sometimes it can be quite difficult to work with CloudForms API. Obviously, there is nothing complicated using it, but if you need to extend API to provide some additional service, you may spend some serious time to understand how it works.
There are several ways to extend existing API, today I will describe the fastest, which takes just a little time to realize.
This method is suitable only if you don't need to add new instances to CloudForms, but just want to collect some existing info in one place and pass it through API.
CloudForms API provides limited information about services, which wasn't enough for our case. We wanted to add some extra info, especially about Virtual Machines.
Firs of all describe new API instance in ${VMDB}/config/api.yml
---
  :extended_services:
    :description: Extended Service API
    :options:
    - :collection
    :methods: *70174834086080
:extended_services: name of our new API
- :collection means that it will be available directly, so our new API URL is {hostname}/api/extended_services
:methods: *70174834086080 is a hash of HTTP Methods available for collection. You can find all values in ${VMDB}/config/api.yml
---
:method:
  :names:
  - :get
  - :put
  - :post
  - :patch
  - :delete
  :sets:
    :g: &70174834086080
    - :get
    :gp: &70174834085860
    - :get
    - :post
    :gpd: &70174834085620
    - :get
    - :post
    - :delete
    :gpppd: &70174834084700
    - :get
    - :put
    - :post
    - :patch
    - :delete

In our case we need only GET method. Though POST method realization is in some way similar to GET, there are still some serious differences, which we will describe in future articles.
Then we need to write our new controller, which will actually do all the work ${VMDB}/app/controllers/api_controller/extended_services.rb
class ApiController
  module ExtendedServices

    def show_extended_services
 #@req[:c_id] is global variable, id of collection instance from url.
  #check if this param exists
 #and if not, render whole collection instead of one element
        if !@req[:c_id]
          render_extended_services_collection()
        else
          render_extended_services(@req[:c_id])
        end
    end

    def render_extended_services_collection()
      resp = Jbuilder.new
      my_services = find_filtered_api(Service, :all)
      resp.resources my_services do |my_service|
        resp.id my_service.id
        resp.name my_service.name
        resp.created_at my_service.created_at
        resp.template my_service.service_template_id
        resp.cpus my_service.aggregate_direct_vm_cpus
        resp.vms my_service.vms.size
        resp.memory my_service.aggregate_all_vm_memory
      end
      render :json => resp.target!
    end

    def  render_extended_services(id)
      my_service = find_by_id_filtered_api(Service, id)
      resp = Jbuilder.new
      resp.resources do
        resp.id id
        resp.name my_service.name
        resp.created_at my_service.created_at
        resp.vms my_service.vms do |vm|
          resp.id vm.id
          resp.name vm.name
          resp.state vm.normalized_state
          resp.cpu vm.num_cpu
          resp.used_storage vm.used_storage
          resp.snapshots vm.snapshots
        end
      end
    render :json => resp.target!
    end
  end
end
Thit is important, that one method called "show_"+{api name} because after /api/extended_services called, CloudForms looks exactly for method with this name and invokes it. Here all information that we need from service is collected and rendered.
All CloudForms api responses should be in JSON, we use Jbuilder to make JSON from our data.
We use custom methods find_filtered_api and find_by_id_filtered_api to find instances, user have access to. This methods are just slightly corrected methods find_filtered and find_by_id_filtered from app/controllers/application_controller.rb. We use basic ideas of this methods, but we apply all filters to @user global variable which stores API user. We save this methods in our custom api helper, so we can use them in future.
${VMDB}app/helpers/api_helper/custom_helper.rb
---
    def find_filtered_api(db, count, options={})
      user     =  User.find_by_userid(@auth_user)
      mfilters = user ? user.get_managed_filters   : []
      bfilters = user ? user.get_belongsto_filters : []
        $api_log.info(bfilters)
      if db.respond_to?(:find_filtered) && !mfilters.empty?
        result = db.find_tags_by_grouping(mfilters, :conditions => options[:conditions], :ns=>"*")
      else
        result = Rbac.search(:class => db, :conditions => options[:conditions], :userid => @auth_user, :results_format => :objects).first
      end

      result = MiqFilter.apply_belongsto_filters(result, bfilters) if db.respond_to?(:find_filtered) &&  result

      result
    end

    def find_by_id_filtered_api(db, id)
      raise "Invalid input" unless is_integer?(id)
      userid  = @auth_user
      unless db.where(:id => from_cid(id)).exists?
        msg = I18n.t("flash.record.selected_item_no_longer_exists", :model => ui_lookup(:model => db.to_s))
        raise msg
      end
      msg = "User '#{userid}' is not authorized to access '#{ui_lookup(:model=>db.to_s)}' record id '#{id}'"
      conditions = ["#{db.table_name}.id = ?", id]
      result = Rbac.search(:class => db, :conditions => conditions, :userid => userid, :results_format => :objects).first.first
      raise msg if result.nil?
      result
    end
---
Include new module in ${VMDB}/app/controllers/api_controller.rb
---
  include_concern "ExtendedServices"
Restart CFME:
$> cd /var/www/miq/vmdb/
$> rake evm:restart
That's it!
Now we can go to our newly created pages and see results
All collection {hostname}/api/extended_services
One service {hostname}/api/extended_services/57

0 comments:

CloudForms UI add new menu button

11:51 , , 1 Comments


During the latest CloudForms integration with external system we have needed to add custom page. But first of all we have to add some custom menu button which will be a reference to custom page.
custom menu button Test Application
You can find all classes responsible for menu loading in ${VMDB}/app/presenters/menu directory.
Here following files:
manager.rbmain class Menu::Manager
section.rbtop-menu buttons (Services, Clouds, Infrastructure, ...)
item.rbmenu buttons (Providers, Clusters, Hosts, ...)
default_menu.rbloads default menu shipped with ManageIQ
custom_loader.rbloads menu buttons from custom YAML files
Create custom YAML file in directory ${VMDB}/product/menubar/
---
# Custom menubar for Test application
# in Infrastructure menu
type: item
parent: inf
id: "test_app"
name: "Test Application"
feature: "test_app"
rbac:
  feature: test_app
href: '/test'
type: item there are currently only 2 supported values:
  •  section
  •  item (whatever not "section" is considered to be item)
parent: inf means it will be added to Infrastructure section (full list in default_menu.rb)
feature: "test_app" defines associated "Product Feature"
rbac: feature: test_app defines list of "Product Features", which make this menu visible. Most of cases it's the same as feature: value.
href: '/test' URL routes to application page

As we specified custom Product Feature "test_app" then we need to create such feature.
Create custom directory ${VMDB}/db/fixtures/miq_product_features/ and create here file feature_test_app.yml.
For example, define view access rules 'List' and 'Show' collapsed with common parent rule 'View'.
---
#Test Application
:name: Test Application
:description: Test Application Views and Access Rules
:feature_type: node
:identifier: test_app
:parent:
:children:
- :name: View 
  :description: View Test Application
  :feature_type: view
  :identifier: test_app_view
  :children:
  - :name: List
    :description: Display Lists of Test Application
    :feature_type: view
    :identifier: test_app_list
  - :name: Show
    :description: Display Individual Items of Test Application
    :feature_type: view
    :identifier: test_app_show
It allows us to configure roles access to this product feature:
configuration of role access: custom product feature
Restart CFME:
$> cd /var/www/miq/vmdb/
$> rake evm:restart
After click on custom button it redirects to specified '/test_app' URL.

TODO: next time custom application page

1 comments: