Charts on dashboard in Manageiq Capablanca : scheme of generating content and fixing problem

20:54 4 Comments

 A lot of people, which use Manageiq Capablanca, faced with problem of charts on dashboard. In this post i will descibe you scheme of building charts and solution of problem with dashboard. Let's go!


Scheme of building content of charts



In generating of contents of charts all functions of model MiqWidget  are involved.
But the most interesting:
-   generate_content
- generate_report
- generate_report_result
In function generate_content, firstly function generate_report is calling and then generate_report_result is calling. So, we have content to build our charts. Then we go to MiqWidget::ChartContent
--
class MiqWidget::ChartContent < MiqWidget::ContentGeneration
  def generate(user_or_group)
    theme = user_or_group.settings.fetch_path(:display, :reporttheme) if user_or_group.kind_of?(User)

    # TODO: default reporttheme to MIQ since it doesn't look like we ever change it
    theme ||= "MIQ"
    report.to_chart(theme, false, MiqReport.graph_options)
    report.chart
  end
end

We can see, that all charts use standard graphic options(size, colours and other).
So, then we should to know more about function to_chart:

--
module MiqReport::Formatters::Graph
  extend ActiveSupport::Concern

  module ClassMethods
  ...
  def to_chart(theme = nil, show_title = false, graph_options = nil)
    ReportFormatter::ReportRenderer.render(Charting.format) do |e|
      e.options.mri           = self
      e.options.show_title    = show_title
      e.options.graph_options = graph_options unless graph_options.nil?
      e.options.theme         = theme
    end

  end
end
Charting.format = jqplot ( for all charts). But what about ReportFormatter::ReportRenderer.render?

--
module ReportFormatter
  class ReportRenderer < Ruport::Controller
    stage :document_header, :document_body, :document_footer
    finalize :document
    options { |o| o.mri = o.show_title = o.theme = o.table_width = o.alignment = o.graph_options = nil }
  end
end

How we can see, ReportFormatter::ReportRenderer use standard library ruport, which use module ChartCommon. And this module, finally, build contents of charts. Look at module ChartCommon and some important functions of it:
-
    def build_document_body
      return no_records_found_chart if mri.table.nil? || mri.table.data.blank?

      # find the highest chart value and set the units accordingly for large disk values (identified by GB in units)
      maxcols = 8
      divider = 1
      if graph_options[:units] == "GB" && !graph_options[:composite]
        maxval = 0
        mri.graph[:columns].each_with_index do |col, col_idx|
          next if col_idx >= maxcols
          newmax = mri.table.data.collect { |r| r[col].nil? ? 0 : r[col] }.sort.last
          maxval = newmax if newmax > maxval
        end
        if maxval > 10.gigabytes
          divider = 1.gigabyte
        elsif maxval > 10.megabytes
          graph_options[:units] = "MB"
          divider = 1.megabyte
        elsif maxval > 10.kilobytes
          graph_options[:units] = "KB"
          divider = 1.kilobyte
        else
          graph_options[:units] = "Bytes"
          graph_options[:decimals] = 0
          divider = 1
        end
        mri.title += " (#{graph_options[:units]})" unless graph_options[:units].blank?
      endfun = case graph_options[:chart_type]
            when :performance then :build_performance_chart # performance chart (time based)
            when :util_ts     then :build_util_ts_chart     # utilization timestamp chart (grouped columns)
            when :planning    then :build_planning_chart    # trend based planning chart
            else                                            # reporting charts
              mri.graph[:mode] == 'values' ? :build_reporting_chart_numeric : :build_reporting_chart
            end
      method(fun).call(maxcols, divider)
    end
    
    def build_reporting_chart_other
      save_key   = nil
      counter    = 0
      categories = []                      # Store categories and series counts in an array of arrays
      mri.table.data.each_with_index do |r, d_idx|
        if d_idx > 0 && save_key != r[mri.sortby[0]]
          save_key = nonblank_or_default(save_key)
          categories.push([save_key, counter])    # Push current category and count onto the array
          counter = 0
        end
        save_key = r[mri.sortby[0]]
        counter += 1
      end
      # add the last key/value to the categories and series arrays
      save_key = nonblank_or_default(save_key)
      categories.push([save_key, counter])        # Push last category and count onto the array

      categories.sort! { |a, b| b.last <=> a.last }
      (keep, show_other) = keep_and_show_other
      if keep < categories.length                      # keep the cathegories w/ highest counts
        other = categories.slice!(keep..-1)
        ocount = other.reduce(0) { |a, e| a + e.last } # sum up and add the other counts
        categories.push(["Other", ocount]) if show_other
      end

      series = categories.each_with_object(
        series_class.new(pie_type? ? :pie : :flat)) do |cat, a|
        a.push(:value => cat.last, :tooltip => "#{cat.first}: #{cat.last}")
      end

      # Pie charts put categories in legend, else in axis labels
      limit = pie_type? ? LEGEND_LENGTH : LABEL_LENGTH
      categories.collect! { |c| slice_legend(c[0], limit) }
      add_axis_category_text(categories)
      add_series(mri.headers[0], series)
    end 

Functions of module ChartCommon form content of charts belonging to MiqWidget, and this content is saved in database in table miq_widgets_contents.
Then, when we want to see at charts on dashboard this content is parsed by YAMl parser and converted in charts. Look at page of charts:

-
-#
  Parameters:
    widget -- MiqWidget instance
- width  ||= 350
- height ||= 250
.panel-body.chart_widget{:style => "text-align: center; padding: 0"}
  .mc{:id => "dd_w#{widget.id}_box",
    :style => @sb[:dashboards][@sb[:active_db]][:minimized].include?(widget.id) ? 'display:none' : ''}

    - if widget.contents_for_user(current_user).contents.blank?
      = _('No chart data found')
      \. . .
    - datum = widget.contents_for_user(current_user).contents
    - if Charting.data_ok?(datum)
      -# we need to count all charts to be able to display multiple
      -# charts on a dashboard screen

      - datum = widget.contents_for_user(current_user).contents
      - WidgetPresenter.chart_data.push(:xml => datum)
      - chart_index = WidgetPresenter.chart_data.length - 1
      - chart_data = YAML.load(datum) if Charting.backend == :jqplot

      = chart_local(chart_data,
                    :id     => "miq_widgetchart_#{chart_index}".html_safe,
                    :width  => width,
                    :height => height)
    - else
      = _('Invalid chart data. Try regenerating the widgets.')
 

Problem with charts and its solution

When you install new version of Manageiq - Capablanca, you can see this on dashboard:


The reason for this problem is that in ChartCommon data is generated
in the form of Hash, but we need YAML.
So, you should add one correction in MiqWidget::ChartContent:

-
class MiqWidget::ChartContent < MiqWidget::ContentGeneration
  def generate(user_or_group)
    theme = user_or_group.settings.fetch_path(:display, :reporttheme) if user_or_group.kind_of?(User)

    # TODO: default reporttheme to MIQ since it doesn't look like we ever change it
    theme ||= "MIQ"
    report.to_chart(theme, false, MiqReport.graph_options)
    report.chart.to_yaml
  end
end

 
Result:


4 comments:

CFME Reports: scheme and extensions

14:20 0 Comments

About scheme of Reports


It's necessary to say in this topic about model MiqExpression, which is stored information about tables, which are used for Reports. This model have four enumerators related with Reports:

1. Enumerator @@base_tables(not fully):
---
@@base_tables = %w{
    Network
    AuditEvent
    AvailabilityZone
    BottleneckEvent
    Chargeback
    CloudResourceQuota
    CloudTenant
    Compliance
    ConfiguredSystemForeman
    ConfigurationManager
    EmsCloud
    EmsCluster
    EmsClusterPerformance
    EmsEvent
    EmsInfra
    ExtManagementSystem
    Flavor
    Host
    Vm
    ...

This enumerator includes names of models, on which reports are built. We can see at snapshot:

2. Enumerator @@include_tables (not fully too):
---
@@include_tables = %w{
    advanced_settings
    audit_events
    availability_zones
    cloud_networks
    cloud_resource_quotas
    cloud_tenants
    compliances
    compliance_details
    configuration_profiles
    configuration_managers
    configured_systems
    customization_scripts
    customization_script_media
    customization_script_ptables
    disks
    networks
    hardware
    ...

This enumerator includes tables, which are not base for reports, but we can see and use these tables, because they are part of @@base_tables. For example, table networks belongs to table hardware, and this table belongs to table of model Vm. So, networks and hardware are included in @@include_tables, and we can see them in available tables for reports. See on snapshot:

 3. Enumerator EXCLUDE_COLUMNS (not fully too):
---
EXCLUDE_COLUMNS = %w{
    ^.*_id$
    ^id$
    ^min_derived_storage.*$
    ^max_derived_storage.*$
    assoc_ids
    capture_interval
    filters
    icon

    intervals_in_rollup

    max_cpu_ready_delta_summation
    max_cpu_system_delta_summation
    max_cpu_used_delta_summation
    max_cpu_wait_delta_summation
    max_derived_cpu_available
    max_derived_cpu_reserved
    max_derived_memory_available
    max_derived_memory_reserved

    memory_usage
    ...

This enumerator includes fields, which should not be displayed in reports. For example, we can't see field password in report, which is based on model EVM User,but  password stores in table of this model.

4. Enumerator EXCLUDE_EXCEPTIONS:
---
EXCLUDE_EXCEPTIONS = %w{
    capacity_profile_1_memory_per_vm_with_min_max
    capacity_profile_1_vcpu_per_vm_with_min_max
    capacity_profile_2_memory_per_vm_with_min_max
    capacity_profile_2_vcpu_per_vm_with_min_max
    chain_id
    guid
  }
 
This enumerator includes fields, which should not be displayed in reports too.

Our extension for Reports


Now, we know scheme of Reports and we can add custom extension.For example:
Our task: We can click on VM in report, which includes VM Name and IP adress, and we can go to this VM. Finally, IP adress should be displayed in base model for reports Perfomance - VMs.
Our actions:

---
class VmPerformance < MetricRollup
  default_scope { where "resource_type = 'VmOrTemplate' and resource_id IS NOT NULL" }

  belongs_to :host,        :foreign_key => :parent_host_id
  belongs_to :ems_cluster, :foreign_key => :parent_ems_cluster_id
  belongs_to :storage,     :foreign_key => :parent_storage_id
  belongs_to :vm,          :foreign_key => :resource_id, :class_name => 'VmOrTemplate'

  virtual_column :ipaddresses,    :type => :string_set

  def ipaddresses
    return self.vm.hardware.ipaddresses
  end

end
- new virtual column ipaddresses with type string_set;
 - definition of ipaddresses (we use vm - model VmPerformance belongs to model of this table).

 Results:

0 comments: