CloudForms API extension: custom model

16:54 0 Comments

Today we study new way to extend CloudForms API. Using this method, we will create new model, which will collect information from other models and transform it as we need. There is not easy way, but it's worth to try.
For example, we will create two models - MonthBilling and AllBilling.
Firstly, we look at model MonthBilling. This model will collect of information for the last month. MonthBilling includes: name of services, number of vms, and cpu usage, memory usage, net usage, disk usage and total cost. This model is based on model MetricRollup and it contains information about accounts for cpu, memory, usage internet, usage disk space. Also, MetricRollup stores information about resources, for which the accounts are and time, during which the accounts are calculated.
 In interface of MonthBilling we will use ActsAsArModel which is provide dynamic AR-like model, where columns defined by hashes.

---
# Model MonthBilling
class MonthBilling < ActsAsArModel

@us = nil

  set_columns_hash(
    :id              => :integer,
    :name            => :string,
    :vms             => :integer,
    :cpu             => :float,
    :mem             => :float,
    :disk            => :float,
    :net             => :float,
    :tco             => :float
  )

  def self.sortable?
    true
  end

  def self.find(*args)
    res = []
    arg = *args
    method = *args
    first_arg = args.first.to_s
    if (first_arg.eql?("all"))
      if (args[1].nil?)
        id_user = User.find_by_userid(@us)
        services = Service.where(evm_owner_id: id_user.id)
        res = ApiController.helpers.add_metrics(services,"month")
      else
        size_conditions = args[1][:conditions].size
        res_string = ""
        for i in 9 .. size_conditions
          if ((args[1][:conditions][i].to_i)== 0 && !(args[1][:conditions][i].eql?("0")))
            break
          end
          res_string+=args[1][:conditions][i]
         end
        if (args[1][:conditions][i].eql?(","))
          res = self.find(:all)
        else
          res_number = res_string.to_i
          res = self.find(res_number)
        end
      end
    else
    id_user = User.find_by_userid(@us)
      services = Service.where(id: args.first.to_i, evm_owner_id: id_user.id)
      res = ApiController.helpers.add_metrics(services,"month")
    end
    res
  end

  def self.auth_initiliaze(user)
  @us = user
  end

  def self.count(*args)
    id_user = User.find_by_userid(@us)
    Service.where(evm_owner_id: id_user.id).count
  end


  def self.scoped(options = {})
    self.find(:all)
  end

end
You have to implement following methods:
 -self.count(*args): this method is needed for quick information about count of billing. Billings are counted for each service, and count of billing is equal to the count of services.
 - self.auth_initiliaze(user): this method is necessary for searching services, which are belong to this user.@us is initialized before, in renderer.rb
- self.find(*args): in this function search is based on arguments. If first  argument - id, then then result is based on this id (service is searched by id). If first  argument - :all, and second argument is null, then result is based on all services and all information about. If first  argument - :all, and second argument is not null, then second argument is parsed, and then result is based on results of parsing(if it is id in second arguments, then result is based on this id. In another case, result is based on all services). Required by ActsAsArModel.
- self.scoped(options = nil): required by RBAC (${VMDB}/app/models/rbac.rb)
- attribute id: it's required to create correct "href"s.
In MonthBilling we will show billings for services belonging to user. So, to know what user work with our API, we will add function self.auth_initiliaze in  ${VMDB}/app/helpers/api_helper/renderer.rb
---
# $VMDB/app/helpers/api_helper/renderer.rb
module ApiHelper
  module Renderer
    #
    # Helper proc for rendering a collection of type specified.
    #
    def render_collection_type(type, id, is_subcollection = false)
      klass = collection_class(type)
      if (klass.respond_to?(:auth_initiliaze))
      klass.auth_initiliaze(@auth_user)
      end
.....
Model MonthBilling uses important function - add_metrics. We save this method in ${VMDB}/app/helpers/api_helper/custom_helper.rb
---
# $VMDB/app/helpers/api_helper/custom_helper.rb
def add_metrics(services,time)
      $api_log.info("[DBG] start IBAHelper.add_metrics()")
      res = []
      required_time = nil
      if (time.to_s.eql?("month"))
        required_time = Time.now - 1.month
      elsif (time.to_s.eql?("week"))
        required_time = Time.now - 1.week
      elsif (time.to_s.eql?("day"))
        required_time = Time.now - 1.day
      end
      array_vms = []
      hash_services = {}
      hash_billings = {}
      hash_billings = Hash.new{|h,k| h[k]=Hash.new(&h.default_proc)}
      services.map{|service|
        hash_billings[service][:cpu] = 0
        hash_billings[service][:mem] = 0
        hash_billings[service][:net] = 0
        hash_billings[service][:disk] = 0
        hash_billings[service][:tco] = 0
        service.vms.map{|vm| array_vms.push(vm.id) }
        service.vms.map{|vm| hash_services[vm.id] = service }
      }
      resources = MetricRollup.select("resource_id, SUM(cpu_usage_rate_average) as cpu_usage_rate_average, SUM(mem_usage_absolute_average)as mem_usage_absolute_average, SUM(disk_usage_rate_average) as disk_usage_rate_average,SUM(net_usage_rate_average) as net_usage_rate_average").where(resource_type: 'VmOrTemplate', capture_interval_name: 'hourly', timestamp: (required_time..Time.now), resource_id: array_vms).group(:resource_id)
      hash_services.each_pair do |key, value|
        resource = resources.where(resource_id: key.to_i)
        resource.each do |element|
          $api_log.info("in element - #{key} => #{value}")
          hash_billings[value][:cpu] += element.cpu_usage_rate_average.to_f
          hash_billings[value][:mem] += element.mem_usage_absolute_average.to_f
          hash_billings[value][:net] += element.net_usage_rate_average.to_f
          hash_billings[value][:disk]+= element.disk_usage_rate_average.to_f
          hash_billings[value][:tco] += element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
        end
      end
      hash_billings.each_pair do |key, value|
        if (time.to_s.eql?("month"))
          $api_log.info("if (time.to_s.eql?(month)) - billinsg")
          res.push(MonthBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        elsif (time.to_s.eql?("week"))
          $api_log.info("if (time.to_s.eql?(week) - billings")
          res.push(WeekBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        elsif (time.to_s.eql?("day"))
          $api_log.info("if (time.to_s.eql?(day)) = billings")
          res.push(DayBilling.new(id: key.id, name: key.name, vms: key.vms.size, cpu: value[:cpu], mem: value[:mem], disk: value[:disk], net: value[:net], tco: value[:tco]))
        end
      end
    $api_log.info("[DBG] end IBAHelper.add_metrics()")
    res
    end
1. Firstly, time is calculated, according to argument time.
2.Then hash of billings is initilizated. Hash contains information about id, name of services and count of vms. Other attributes are zero. Also, information about necessary id of vms is collected.
3. Information about cpu, memory and etc. is stored from model MetricRollup, where id = id of vm, capture_interval - hourly, and information is collected for the last month.In hash of billings information about cpu, memory and etc. is summed.
4. According to argument time, new MonthBilling is created. As you can see, according to argument time, you can create other model like this, for example WeekBilling. And, you should to desribe model in ${VMDB}/config/api.yml 
---
# $VMDB/config/api.yml
:billings_monthly:
    :description: Billings
    :options:
    - :collection
    :methods: *70174834086080
    :klass: MonthBilling
Then we will restart CFME:

$> cd /var/www/miq/vmdb/
$> rake evm:restart
And now you can see result.
Go to the page: {hostname}/api/billings_monthly
Output: 
{"name":"billings_monthly","count":5,"subcount":5,"resources":[{"href":"https://{hostname}/api/billings_monthly/12"},{"href":"https://{hostname}/api/billings_monthly/13"},{"href":"https://{hostname}/api/billings_monthly/8"},{"href":"https://{hostname}/api/billings_monthly/11"},{"href":"https://{hostname}/api/billings_monthly/10"}]}


Let's go back to AllBilling. This model is based on collection of information for the last month per hour. This model is subcollection, and information in this model is based on one service. This model includes: name of service, number of vms, and cpu usage, memory usage, net usage, disk usage and total cost. 

---
# $VMDB/app/models/all_billing.rb

class AllBilling < ActsAsArModel

@us = nil

  set_columns_hash(
    :id              => :integer,
    :name            => :string,
    :vms             => :integer,
    :cpu             => :float,
    :mem             => :float,
    :disk            => :float,
    :net             => :float,
    :tco             => :float
  )

  def self.find(*args)
    arg = *args
    method = *args
    first_arg = args.first.to_s
    res = []
    if (first_arg.eql?("all"))
      index = 0
      if (args[1].nil?)
        id_user = User.find_by_userid(@us)
        services = Service.where(evm_owner_id: id_user.id)
        res = ApiController.helpers.add_billings(services)
      else
        size_conditions = args[1][:conditions].size
        res_string = ""
        for i in 9 .. size_conditions
          if ((args[1][:conditions][i].to_i)== 0 && !(args[1][:conditions][i].eql?("0")))
            break
          end
          res_string+=args[1][:conditions][i]
        end
        if (args[1][:conditions][i].eql?(","))
          res = self.find(:all)
        else
          res_number = res_string.to_i
          res = self.find(res_number)
        end
      end
    else
      first_arg = args.first
      res = ApiController.helpers.add_billings_by_id(args.first)
    end
    res
  end


  def self.auth_initiliaze(user)
  @us = user
  end

  def self.count(*args)
    $api_log.info("[DBG] start self.count()")
    0
  end


  def self.scoped(options = {})
   $api_log.info("[DEBUG] (Billings) self.scoped("+options.inspect+")")
   self.find(:all)
  end

end 
 
As this is subcollection, we need to add module Billing in file billings.rb in ${VMDB}/app/controllers/api_controller. When we want to see subcollection, results are created by function {name_module}__query_resource
---
# $VMDB/app/controllers/api_controllers/billings.rb
class ApiController
  module Billings
    #
    # Rates Subcollection Supporting Methods
    #
    def billings_query_resource(data)
      $api_log.info("billings_query_resource")
      data ? ApiController.helpers.add_billings_all(data.id) : {}
    end
  end
end

And we will include this module in ApiController(${VMDB}/app/controller/api_controller.rb)
---
# $VMDB/app/controllers/api_controller.rb
include_concern 'Billings'
Almost all methods are described before. And what about add_billings and add_billings_by_id and add_billings_all. These methods are in ${VMDB}/app/helpers/api_helper/custom_helper.rb
---
# $VMDB/app/helpers/api_helper/custom_helper.rb
 
 def add_billings_all(add_id)
    services = Service.where(id: add_id)
    res = []
    array_vms = []
    hash_billings = {}
    hash_services = {}
    $api_log.info("#{services}")
    services.map{|service|
      service.vms.map{|vm| array_vms.push(vm.id) }
      service.vms.map{|vm| hash_services[vm.id] = service }
      }
     resources = MetricRollup.select("id, resource_id, cpu_usage_rate_average, mem_usage_absolute_average,disk_usage_rate_average,net_usage_rate_average").where(resource_id: array_vms, capture_interval_name: 'hourly').group(:id, :resource_id)
     resources.each do |element|
       hash_billings[:id] =  element.id
       hash_billings[:name] = (hash_services.fetch(element.resource_id)).name
       hash_billings[:vms] = (hash_services.fetch(element.resource_id)).vms.size
       hash_billings[:cpu] = element.cpu_usage_rate_average.to_f
       hash_billings[:mem] = element.mem_usage_absolute_average.to_f
       hash_billings[:net] = element.net_usage_rate_average.to_f
       hash_billings[:disk]= element.disk_usage_rate_average.to_f
       hash_billings[:tco] = element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
       res.push(AllBilling.new(id: hash_billings[:id], name: hash_billings[:name], vms: hash_billings[:vms], cpu: hash_billings[:cpu], mem: hash_billings[:mem], disk: hash_billings[:disk], net: hash_billings[:net], tco: hash_billings[:tco]))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
   end

def add_billings(services)
    res = []
   array_vms = []
   hash_services = {}
   hash_billings = {}
   hash_billings = Hash.new{|h,k| h[k]=Hash.new(&h.default_proc)}
   services.map{|service|
     service.vms.map{|vm| array_vms.push(vm.id) }
     service.vms.map{|vm| hash_services[vm.id] = service }
   }
   resources = MetricRollup.select("id, resource_id,  cpu_usage_rate_average, mem_usage_absolute_average, disk_usage_rate_average, net_usage_rate_average").where(resource_type: 'VmOrTemplate', capture_interval_name: 'hourly', timestamp: ((Time.now - 1.month)..Time.now), resource_id: array_vms).group(:id, :resource_id)
     resources.each do |element|
       hash_billings[:id] =  element.id
       hash_billings[:name] = (hash_services.fetch(element.resource_id)).name
       hash_billings[:vms] = (hash_services.fetch(element.resource_id)).vms.size
       hash_billings[:cpu] = element.cpu_usage_rate_average.to_f
       hash_billings[:mem] = element.mem_usage_absolute_average.to_f
       hash_billings[:net] = element.net_usage_rate_average.to_f
       hash_billings[:disk]= element.disk_usage_rate_average.to_f
       hash_billings[:tco] = element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.net_usage_rate_average.to_f + element.disk_usage_rate_average.to_f
       res.push(AllBilling.new(id: hash_billings[:id], name: hash_billings[:name], vms: hash_billings[:vms], cpu: hash_billings[:cpu], mem: hash_billings[:mem], disk: hash_billings[:disk], net: hash_billings[:net], tco: hash_billings[:tco]))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
end

  def add_billings_by_id(add_id)
    res = []
    serv = nil
    services = Service.find(:all)
    resources = MetricRollup.select("id, resource_id, cpu_usage_rate_average, mem_usage_absolute_average,disk_usage_rate_average,net_usage_rate_average").where(id: add_id).group(:id, :resource_id)
    resources.each do |element|
      services.each do |service|
        service.vms.each do |vm|
          if (vm.id == element.resource_id)
          serv = service
          end
        end
      end
     res.push(AllBilling.new(id: element.id, name: serv.name, vms: serv.vms.size, cpu: element.cpu_usage_rate_average.to_f, mem: element.mem_usage_absolute_average.to_f, disk: element.disk_usage_rate_average.to_f, net: element.net_usage_rate_average.to_f, tco: (element.cpu_usage_rate_average.to_f + element.mem_usage_absolute_average.to_f + element.disk_usage_rate_average.to_f + element.net_usage_rate_average.to_f)))
     end
     $api_log.info("[DBG] end IBAHelper.add_billings()")
    res
  end
Methods add_billings_all and add_billings are based on the same things:
1. Information about services and id of vms is collected in hash_services and array_vms.
2. Information collected from MetricRollup by id of vms.
3. All information about billings are in hash_billings.
4. New AllBilling is based on information of hash_billings.
In method add_billings_by_id we have id of element from MetricRollup and we will find needed element and service.
Then, we should describe this subcollection in ${VMDB}/config/api.yml
---
# $VMDB/config/api.yml

:billings:
    :description: Billings
    :options:
    - :subcollection
    :methods: *70174834086080
    :klass: AllBilling
 
:services:
    :description: Services
    :options:
    - :collection
    :methods: *70174834084700
    :klass: Service
    :subcollections:
    - :billings
    - :tags
Restart CFME:

$> cd /var/www/miq/vmdb/
$> rake evm:restart
It's time to see results.
Go to the page: https://{hostname}.128/api/services/8/billings/
Output: 
{"name":"billings","count":0,"subcount":128,"resources":[{"href":"https://{hostname}/api/services/8/billings/368"},{"href":"https://{hostname}/api/services/8/billings/780"}]}
Go to the page: https://{hostname}.128/api/services/8/billings/368
Output: 
{"href":"https://{hostname}/api/services/8/billings/368","id":368,"name":"AnnKurnik-20150918-070008","vms":1,"cpu":1.39155555555556,
"mem":0.871333333333333,"disk":0.0,
"net":10.1222222222222,"tco":12.385111111111094}

0 comments:

CloudForms API extension: fast way

17:41 0 Comments

Sometimes it can be quite difficult to work with CloudForms API. Obviously, there is nothing complicated using it, but if you need to extend API to provide some additional service, you may spend some serious time to understand how it works.
There are several ways to extend existing API, today I will describe the fastest, which takes just a little time to realize.
This method is suitable only if you don't need to add new instances to CloudForms, but just want to collect some existing info in one place and pass it through API.
CloudForms API provides limited information about services, which wasn't enough for our case. We wanted to add some extra info, especially about Virtual Machines.
Firs of all describe new API instance in ${VMDB}/config/api.yml 
---
  :extended_services:
    :description: Extended Service API
    :options:
    - :collection
    :methods: *70174834086080
:extended_services: name of our new API
- :collection means that it will be available directly, so our new API URL is {hostname}/api/extended_services
:methods: *70174834086080 is a hash of HTTP Methods available for collection. You can find all values in ${VMDB}/config/api.yml 
---
:method:
  :names:
  - :get
  - :put
  - :post
  - :patch
  - :delete
  :sets:
    :g: &70174834086080
    - :get
    :gp: &70174834085860
    - :get
    - :post
    :gpd: &70174834085620
    - :get
    - :post
    - :delete
    :gpppd: &70174834084700
    - :get
    - :put
    - :post
    - :patch
    - :delete
In our case we need only GET method. Though POST method realization is in some way similar to GET, there are still some serious differences, which we will describe in future articles.
Then we need to write our new controller, which will actually do all the work ${VMDB}/app/controllers/api_controller/extended_services.rb 
class ApiController
  module ExtendedServices

    def show_extended_services
 #@req[:c_id] is global variable, id of collection instance from url.
  #check if this param exists
 #and if not, render whole collection instead of one element
        if !@req[:c_id]
          render_extended_services_collection()
        else
          render_extended_services(@req[:c_id])
        end
    end

    def render_extended_services_collection()
      resp = Jbuilder.new
      my_services = find_filtered_api(Service, :all)
      resp.resources my_services do |my_service|
        resp.id my_service.id
        resp.name my_service.name
        resp.created_at my_service.created_at
        resp.template my_service.service_template_id
        resp.cpus my_service.aggregate_direct_vm_cpus
        resp.vms my_service.vms.size
        resp.memory my_service.aggregate_all_vm_memory
      end
      render :json => resp.target!
    end

    def  render_extended_services(id)
      my_service = find_by_id_filtered_api(Service, id)
      resp = Jbuilder.new
      resp.resources do
        resp.id id
        resp.name my_service.name
        resp.created_at my_service.created_at
        resp.vms my_service.vms do |vm|
          resp.id vm.id
          resp.name vm.name
          resp.state vm.normalized_state
          resp.cpu vm.num_cpu
          resp.used_storage vm.used_storage
          resp.snapshots vm.snapshots
        end
      end
    render :json => resp.target!
    end
  end
end
Thit is important, that one method called "show_"+{api name} because after /api/extended_services called, CloudForms looks exactly for method with this name and invokes it. Here all information that we need from service is collected and rendered.
All CloudForms api responses should be in JSON, we use Jbuilder to make JSON from our data.
We use custom methods find_filtered_api and find_by_id_filtered_api to find instances, user have access to. This methods are just slightly corrected methods find_filtered and find_by_id_filtered from app/controllers/application_controller.rb. We use basic ideas of this methods, but we apply all filters to @user global variable which stores API user. We save this methods in our custom api helper, so we can use them in future.
${VMDB}app/helpers/api_helper/custom_helper.rb 
---
    def find_filtered_api(db, count, options={})
      user     =  User.find_by_userid(@auth_user)
      mfilters = user ? user.get_managed_filters   : []
      bfilters = user ? user.get_belongsto_filters : []
        $api_log.info(bfilters)
      if db.respond_to?(:find_filtered) && !mfilters.empty?
        result = db.find_tags_by_grouping(mfilters, :conditions => options[:conditions], :ns=>"*")
      else
        result = Rbac.search(:class => db, :conditions => options[:conditions], :userid => @auth_user, :results_format => :objects).first
      end

      result = MiqFilter.apply_belongsto_filters(result, bfilters) if db.respond_to?(:find_filtered) &&  result

      result
    end

    def find_by_id_filtered_api(db, id)
      raise "Invalid input" unless is_integer?(id)
      userid  = @auth_user
      unless db.where(:id => from_cid(id)).exists?
        msg = I18n.t("flash.record.selected_item_no_longer_exists", :model => ui_lookup(:model => db.to_s))
        raise msg
      end
      msg = "User '#{userid}' is not authorized to access '#{ui_lookup(:model=>db.to_s)}' record id '#{id}'"
      conditions = ["#{db.table_name}.id = ?", id]
      result = Rbac.search(:class => db, :conditions => conditions, :userid => userid, :results_format => :objects).first.first
      raise msg if result.nil?
      result
    end
---
Include new module in ${VMDB}/app/controllers/api_controller.rb 
---
  include_concern "ExtendedServices"
Restart CFME: 
$> cd /var/www/miq/vmdb/
$> rake evm:restart
That's it!
Now we can go to our newly created pages and see results
All collection {hostname}/api/extended_services
One service {hostname}/api/extended_services/57

0 comments:

CloudForms UI add new menu button

11:51 , , 1 Comments


During the latest CloudForms integration with external system we have needed to add custom page. But first of all we have to add some custom menu button which will be a reference to custom page.
custom menu button Test Application
You can find all classes responsible for menu loading in ${VMDB}/app/presenters/menu directory.
Here following files:
manager.rbmain class Menu::Manager
section.rbtop-menu buttons (Services, Clouds, Infrastructure, ...)
item.rbmenu buttons (Providers, Clusters, Hosts, ...)
default_menu.rbloads default menu shipped with ManageIQ
custom_loader.rbloads menu buttons from custom YAML files
Create custom YAML file in directory ${VMDB}/product/menubar/ 
---
# Custom menubar for Test application
# in Infrastructure menu
type: item
parent: inf
id: "test_app"
name: "Test Application"
feature: "test_app"
rbac:
  feature: test_app
href: '/test'
type: item there are currently only 2 supported values:
  •  section
  •  item (whatever not "section" is considered to be item)
parent: inf means it will be added to Infrastructure section (full list in default_menu.rb)
feature: "test_app" defines associated "Product Feature"
rbac: feature: test_app defines list of "Product Features", which make this menu visible. Most of cases it's the same as feature: value.
href: '/test' URL routes to application page

As we specified custom Product Feature "test_app" then we need to create such feature.
Create custom directory ${VMDB}/db/fixtures/miq_product_features/ and create here file feature_test_app.yml.
For example, define view access rules 'List' and 'Show' collapsed with common parent rule 'View'. 
---
#Test Application
:name: Test Application
:description: Test Application Views and Access Rules
:feature_type: node
:identifier: test_app
:parent:
:children:
- :name: View 
  :description: View Test Application
  :feature_type: view
  :identifier: test_app_view
  :children:
  - :name: List
    :description: Display Lists of Test Application
    :feature_type: view
    :identifier: test_app_list
  - :name: Show
    :description: Display Individual Items of Test Application
    :feature_type: view
    :identifier: test_app_show
It allows us to configure roles access to this product feature:
configuration of role access: custom product feature
Restart CFME: 
$> cd /var/www/miq/vmdb/
$> rake evm:restart
After click on custom button it redirects to specified '/test_app' URL.

TODO: next time custom application page

1 comments:

Subscribe to: Comments (Atom)