CloudForms API extension: custom model

16:54 0 Comments

Today we study new way to extend CloudForms API. Using this method, we will create new model, which will collect information from other models and transform it as we need. There is not easy way, but it's worth to try.
For example, we will create two models - MonthBilling and AllBilling.
Firstly, we look at model MonthBilling. This model will collect of information for the last month. MonthBilling includes: name of services, number of vms, and cpu usage, memory usage, net usage, disk usage and total cost. This model is based on model MetricRollup and it contains information about accounts for cpu, memory, usage internet, usage disk space. Also, MetricRollup stores information about resources, for which the accounts are and time, during which the accounts are calculated.
 In interface of MonthBilling we will use ActsAsArModel which is provide dynamic AR-like model, where columns defined by hashes.

---
# Model MonthBilling
class MonthBilling < ActsAsArModel

@us = nil

  set_columns_hash(
    :id              => :integer,
    :name            => :string,
    :vms             => :integer,
    :cpu             => :float,
    :mem             => :float,
    :disk            => :float,
    :net             => :float,
    :tco             => :float
  )

  def self.sortable?
    true
  end

  def self.find(*args)
    res = []
    arg = *args
    method = *args
    first_arg = args.first.to_s
    if (first_arg.eql?("all"))
      if (args[1].nil?)
        id_user = User.find_by_userid(@us)
        services = Service.where(evm_owner_id: id_user.id)
        res = ApiController.helpers.add_metrics(services,"month")
      else
        size_conditions = args[1][:conditions].size
        res_string = ""
        for i in 9 .. size_conditions
          if ((args[1][:conditions][i].to_i)== 0 && !(args[1][:conditions][i].eql?("0")))
            break
          end
          res_string+=args[1][:conditions][i]
         end
        if (args[1][:conditions][i].eql?(","))
          res = self.find(:all)
        else
          res_number = res_string.to_i
          res = self.find(res_number)
        end
      end
    else
    id_user = User.find_by_userid(@us)
      services = Service.where(id: args.first.to_i, evm_owner_id: id_user.id)
      res = ApiController.helpers.add_metrics(services,"month")
    end
    res
  end

  def self.auth_initiliaze(user)
  @us = user
  end

  def self.count(*args)
    id_user = User.find_by_userid(@us)
    Service.where(evm_owner_id: id_user.id).count
  end


  def self.scoped(options = {})
    self.find(:all)
  end

end
You have to implement following methods:
 -self.count(*args): this method is needed for quick information about count of billing. Billings are counted for each service, and count of billing is equal to the count of services.
 - self.auth_initiliaze(user): this method is necessary for searching services, which are belong to this user.@us is initialized before, in renderer.rb
- self.find(*args): in this function search is based on arguments. If first  argument - id, then then result is based on this id (service is searched by id). If first  argument - :all, and second argument is null, then result is based on all services and all information about. If first  argument - :all, and second argument is not null, then second argument is parsed, and then result is based on results of parsing(if it is id in second arguments, then result is based on this id. In another case, result is based on all services). Required by ActsAsArModel.
- self.scoped(options = nil): required by RBAC (${VMDB}/app/models/rbac.rb)
- attribute id: it's required to create correct "href"s.
In MonthBilling we will show billings for services belonging to user. So, to know what user work with our API, we will add function self.auth_initiliaze in  ${VMDB}/app/helpers/api_helper/renderer.rb
---
# $VMDB/app/helpers/api_helper/renderer.rb
module ApiHelper
  module Renderer
    #
    # Helper proc for rendering a collection of type specified.
    #
    def render_collection_type(type, id, is_subcollection = false)
      klass = collection_class(type)
      if (klass.respond_to?(:auth_initiliaze))
      klass.auth_initiliaze(@auth_user)
      end
.....
Model MonthBilling uses important function - add_metrics. We save this method in ${VMDB}/app/helpers/api_helper/custom_helper.rb
---
# $VMDB/app/helpers/api_helper/custom_helper.rb
def add_metrics(services,time)
      $api_log.info("[DBG] start IBAHelper.add_metrics()")
      res = []
      required_time = nil
      if (time.to_s.eql?("month"))
        required_time = Time.now - 1.month
      elsif (time.to_s.eql?("week"))
        required_time = Time.now - 1.week
      elsif (time.to_s.eql?("day"))
        required_time = Time.now - 1.day
      end
      array_vms = []
      hash_services = {}
      hash_billings = {}
      hash_billings = Hash.new{|h,k| h[k]=Hash.new(&h.default_proc)}
      services.map{|service|
        hash_billings[service][:cpu] = 0
        hash_billings[service][:mem] = 0
        hash_billings[service][:net] = 0
        hash_billings[service][:disk] = 0
        hash_billings[service][:tco] = 0
        service.vms.map{|vm| array_vms.push(vm.id) }
        service.vms.map{|vm| hash_services[vm.id] = service }
      }
      resources = MetricRollup.select("resource_id, SUM(cpu_usage_rate_average) as cpu_usage_rate_average, SUM(mem_usage_absolute_average)as mem_usage_absolute_average, SUM(disk_usage_rate_average) as disk_usage_rate_average,SUM(net_usage_rate_average) as net_usage_rate_average").where(resource_type: 'VmOrTemplate', capture_interval_name: 'hourly', timestamp: (required_time..Time.now), resource_id: array_vms).group(:resource_id)
      hash_services.each_pair do |key, value|
        resource = resources.where(resource_id: key.to_i)
        resource.each do |element|
          $api_log.info("in element - #{key} => #{value}")
          hash_billings[value][:cpu] += element.cpu_usage_rate_average.to_f
          hash_billings[value][:mem] += element.mem_usage_absolute_average.to_f
          hash_billings[value][:net] += element.net_usage_rate_average.to_f
          hash_billings[value][:disk]+= element.disk_usage_rate_average.to_f
          hash_billings[value][:tco] += element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
        end
      end
      hash_billings.each_pair do |key, value|
        if (time.to_s.eql?("month"))
          $api_log.info("if (time.to_s.eql?(month)) - billinsg")
          res.push(MonthBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        elsif (time.to_s.eql?("week"))
          $api_log.info("if (time.to_s.eql?(week) - billings")
          res.push(WeekBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        elsif (time.to_s.eql?("day"))
          $api_log.info("if (time.to_s.eql?(day)) = billings")
          res.push(DayBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        end
      end
    $api_log.info("[DBG] end IBAHelper.add_metrics()")
    res
    end
1. Firstly, time is calculated, according to argument time.
2.Then hash of billings is initilizated. Hash contains information about id, name of services and count of vms. Other attributes are zero. Also, information about necessary id of vms is collected.
3. Information about cpu, memory and etc. is stored from model MetricRollup, where id = id of vm, capture_interval - hourly, and information is collected for the last month.In hash of billings information about cpu, memory and etc. is summed.
4. According to argument time, new MonthBilling is created. As you can see, according to argument time, you can create other model like this, for example WeekBilling. And, you should to desribe model in ${VMDB}/config/api.yml 
---
# $VMDB/config/api.yml
:billings_monthly:
    :description: Billings
    :options:
    - :collection
    :methods: *70174834086080
    :klass: MonthBilling
Then we will restart CFME:

$> cd /var/www/miq/vmdb/
$> rake evm:restart
And now you can see result.
Go to the page: {hostname}/api/billings_monthly
Output: 
{"name":"billings_monthly","count":5,"subcount":5,"resources":[{"href":"https://{hostname}/api/billings_monthly/12"},{"href":"https://{hostname}/api/billings_monthly/13"},{"href":"https://{hostname}/api/billings_monthly/8"},{"href":"https://{hostname}/api/billings_monthly/11"},{"href":"https://{hostname}/api/billings_monthly/10"}]}


Let's go back to AllBilling. This model is based on collection of information for the last month per hour. This model is subcollection, and information in this model is based on one service. This model includes: name of service, number of vms, and cpu usage, memory usage, net usage, disk usage and total cost. 

---
# $VMDB/app/models/all_billing.rb

class AllBilling < ActsAsArModel

@us = nil

  set_columns_hash(
    :id              => :integer,
    :name            => :string,
    :vms             => :integer,
    :cpu             => :float,
    :mem             => :float,
    :disk            => :float,
    :net             => :float,
    :tco             => :float
  )

  def self.find(*args)
    arg = *args
    method = *args
    first_arg = args.first.to_s
    res = []
    if (first_arg.eql?("all"))
      index = 0
      if (args[1].nil?)
        id_user = User.find_by_userid(@us)
        services = Service.where(evm_owner_id: id_user.id)
        res = ApiController.helpers.add_billings(services)
      else
        size_conditions = args[1][:conditions].size
        res_string = ""
        for i in 9 .. size_conditions
          if ((args[1][:conditions][i].to_i)== 0 && !(args[1][:conditions][i].eql?("0")))
            break
          end
          res_string+=args[1][:conditions][i]
        end
        if (args[1][:conditions][i].eql?(","))
          res = self.find(:all)
        else
          res_number = res_string.to_i
          res = self.find(res_number)
        end
      end
    else
      first_arg = args.first
      res = ApiController.helpers.add_billings_by_id(args.first)
    end
    res
  end


  def self.auth_initiliaze(user)
  @us = user
  end

  def self.count(*args)
    $api_log.info("[DBG] start self.count()")
    0
  end


  def self.scoped(options = {})
   $api_log.info("[DEBUG] (Billings) self.scoped("+options.inspect+")")
   self.find(:all)
  end

end 
 
As this is subcollection, we need to add module Billing in file billings.rb in ${VMDB}/app/controllers/api_controller. When we want to see subcollection, results are created by function {name_module}__query_resource
---
# $VMDB/app/controllers/api_controllers/billings.rb
class ApiController
  module Billings
    #
    # Rates Subcollection Supporting Methods
    #
    def billings_query_resource(data)
      $api_log.info("billings_query_resource")
      data ? ApiController.helpers.add_billings_all(data.id) : {}
    end
  end
end

And we will include this module in ApiController(${VMDB}/app/controller/api_controller.rb)
---
# $VMDB/app/controllers/api_controller.rb
include_concern 'Billings'
Almost all methods are described before. And what about add_billings and add_billings_by_id and add_billings_all. These methods are in ${VMDB}/app/helpers/api_helper/custom_helper.rb
---
# $VMDB/app/helpers/api_helper/custom_helper.rb
 
 def add_billings_all(add_id)
    services = Service.where(id: add_id)
    res = []
    array_vms = []
    hash_billings = {}
    hash_services = {}
    $api_log.info("#{services}")
    services.map{|service|
      service.vms.map{|vm| array_vms.push(vm.id) }
      service.vms.map{|vm| hash_services[vm.id] = service }
      }
     resources = MetricRollup.select("id, resource_id, cpu_usage_rate_average, mem_usage_absolute_average,disk_usage_rate_average,net_usage_rate_average").where(resource_id: array_vms, capture_interval_name: 'hourly').group(:id, :resource_id)
     resources.each do |element|
       hash_billings[:id] =  element.id
       hash_billings[:name] = (hash_services.fetch(element.resource_id)).name
       hash_billings[:vms] = (hash_services.fetch(element.resource_id)).vms.size
       hash_billings[:cpu] = element.cpu_usage_rate_average.to_f
       hash_billings[:mem] = element.mem_usage_absolute_average.to_f
       hash_billings[:net] = element.net_usage_rate_average.to_f
       hash_billings[:disk]= element.disk_usage_rate_average.to_f
       hash_billings[:tco] = element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
       res.push(AllBilling.new(id: hash_billings[:id], name: hash_billings[:name], vms: hash_billings[:vms], cpu: hash_billings[:cpu], mem: hash_billings[:mem], disk: hash_billings[:disk], net: hash_billings[:net], tco: hash_billings[:tco]))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
   end

def add_billings(services)
    res = []
   array_vms = []
   hash_services = {}
   hash_billings = {}
   hash_billings = Hash.new{|h,k| h[k]=Hash.new(&h.default_proc)}
   services.map{|service|
     service.vms.map{|vm| array_vms.push(vm.id) }
     service.vms.map{|vm| hash_services[vm.id] = service }
   }
   resources = MetricRollup.select("id, resource_id,  cpu_usage_rate_average, mem_usage_absolute_average, disk_usage_rate_average, net_usage_rate_average").where(resource_type: 'VmOrTemplate', capture_interval_name: 'hourly', timestamp: ((Time.now - 1.month)..Time.now), resource_id: array_vms).group(:id, :resource_id)
     resources.each do |element|
       hash_billings[:id] =  element.id
       hash_billings[:name] = (hash_services.fetch(element.resource_id)).name
       hash_billings[:vms] = (hash_services.fetch(element.resource_id)).vms.size
       hash_billings[:cpu] = element.cpu_usage_rate_average.to_f
       hash_billings[:mem] = element.mem_usage_absolute_average.to_f
       hash_billings[:net] = element.net_usage_rate_average.to_f
       hash_billings[:disk]= element.disk_usage_rate_average.to_f
       hash_billings[:tco] = element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
       res.push(AllBilling.new(id: hash_billings[:id], name: hash_billings[:name], vms: hash_billings[:vms], cpu: hash_billings[:cpu], mem: hash_billings[:mem], disk: hash_billings[:disk], net: hash_billings[:net], tco: hash_billings[:tco]))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
end

  def add_billings_by_id(add_id)
    res = []
    serv = nil
    services = Service.find(:all)
    resources = MetricRollup.select("id, resource_id, cpu_usage_rate_average, mem_usage_absolute_average,disk_usage_rate_average,net_usage_rate_average").where(id: add_id).group(:id, :resource_id)
    resources.each do |element|
      services.each do |service|
        service.vms.each do |vm|
          if (vm.id == element.resource_id)
          serv = service
          end
        end
      end
     res.push(AllBilling.new(id: element.id, name: serv.name, vms: serv.vms.size, cpu: element.cpu_usage_rate_average.to_f, mem: element.mem_usage_absolute_average.to_f, disk: element.disk_usage_rate_average.to_f, net: element.net_usage_rate_average.to_f, tco: (element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.disk_usage_rate_average.to_f + element.net_usage_rate_average.to_f)))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
  end
Methods add_billings_all and add_billings are based on the same things:
1. Information about services and id of vms is collected in hash_services and array_vms.
2. Information collected from MetricRollup by id of vms.
3. All information about billings are in hash_billings.
4. New AllBilling is based on information of hash_billings.
In method add_billings_by_id we have id of element from MetricRollup and we will find needed element and service.
Then, we should describe this subcollection in ${VMDB}/config/api.yml
---
# $VMDB/config/api.yml

:billings:
    :description: Billings
    :options:
    - :subcollection
    :methods: *70174834086080
    :klass: AllBilling
 
:services:
    :description: Services
    :options:
    - :collection
    :methods: *70174834084700
    :klass: Service
    :subcollections:
    - :billings
    - :tags
Restart CFME:

$> cd /var/www/miq/vmdb/
$> rake evm:restart
It's time to see results.
Go to the page: https://{hostname}.128/api/services/8/billings/
Output: 
{"name":"billings","count":0,"subcount":128,"resources":[{"href":"https://{hostname}/api/services/8/billings/368"},{"href":"https://{hostname}/api/services/8/billings/780"}]}
Go to the page: https://{hostname}.128/api/services/8/billings/368
Output: 
{"href":"https://{hostname}/api/services/8/billings/368","id":368,"name":"AnnKurnik-20150918-070008","vms":1,"cpu":1.39155555555556,
"mem":0.871333333333333,"disk":0.0,
"net":10.1222222222222,"tco":12.385111111111094}

Unknown

IBA Group, Minsk

0 comments:

Subscribe to: Post Comments (Atom)