Nextcloud and Keycloak (SAML)


Preamble: we've started Single-Sign On implementation between our products, the ManageIQ supports SSO using SAML v2.0 and there is good documentation on it using Keycloak authentication server.
I did not find any SSO examples for Nextcloud and Keycloak with SAML protocol, so that's why I published this article.
Tip: if you are going integrate Nextcloud with Shibboleth - see https://help.nextcloud.com/t/how-does-the-user-saml-app-work/1419/5


Assume that you have installed Keycloak server and configured user authentication provider (we use Domino LDAP for this purpose).
Also assume that you have installed and configured Nextcloud and can login with admin account.

Step 1. Activate user_saml plugin in nextcloud


Hint: If you failed to configure SAML correctly - you can not login even with admin account. So I recommend to use two different browsers - stay logged as admin in one browser and test SAML login in another one. Alternatively if you failed to login as admin you can disable user_saml app from command line and login back as usual admin account.

sudo -u apache php console.php app:disable user_saml

Step 2. Go to "Admin -> SAML Authentication"


You have to fill following form to configure SAML on Nextcloud and to generate Metadata XML, which then will be imported to Keycloak server.


Some terms used in this form:

  • Service Provider (SP) - Nextcloud.
  • Identity Provider (IdP) - Keycloak.

SP and IdP exchanges information with XML messages, some part of this information can be encrypted and signed with private keys and certificates (!!! it's not related to HTTPS/SSL - it's another certificate).

Step 3. Configure user map attribute (user property) 

email - attribute which will be used to map user ID, we use email, but in your case it can be something else, like uid or username

Step 4. Generate certificate and private key on SP (Nextcloud)

From SP side you can provide X.509 certificate and Private key and sign XML messages with them, so lets generate it:

cd /var/www/html/nc
mkdir certs
cd certs
openssl req -newkey rsa:2048 -nodes -keyout user_saml.key \
        -x509 -days 1024 -out user_saml.crt

Click "ServiceProvider settings" link and copy content of files user_saml.crt to user_saml.key to the first and second field appropriately.

Step 5. Configure IdP (Keycloak) URL

Fortunately, Keycloak use only one URL endpoint for all  SAML requests:

http(s)://{host}:{port}/auth/realms/{realm}/protocol/saml

In Keycloak we use realm named 'master':


In our case specify in all three URL fields: http://172.20.140.174:8080/auth/realms/master/protocol/saml

Step 6. IdP (Keycloak) realm certificate and security settings

Copy it from: Realm settings -> Keys -> All -> Certificate

Configure Security Settings:

  • Offered:
    • Indicates whether the <samlp:AuthnRequest> messages sent by this SP will be signed. [Metadata of the SP will offer this info]
    • Indicates whether the <samlp:logoutRequest> messages sent by this SP will be signed.
    • Indicates whether the <samlp:logoutResponse> messages sent by this SP will be signed.
  • Required:
    • Indicates a requirement for the <samlp:Response>, <samlp:LogoutRequest> and <samlp:LogoutResponse> elements received by this SP to be signed.
    • Indicates a requirement for the <saml:Assertion> elements received by this SP to be signed. [Metadata of the SP will offer this info]
    • Indicates a requirement for the <saml:Assertion> elements received by this SP to be encrypted. (Update: Have some issue here, so could not configure Assertion encryption properly)


Step 7. Configure Keycloak client


If everything is ok, you will see green label "Metadata valid". Download it using nearby button.
It should looks like: https://drive.google.com/open?id=0B3eCoRSksDuBb25BQVRjY2g1ekk

Nextcloud is a client for Keycloak Identity Provider, so on Keycloak server "Clients -> Create -> Import: Select File", select downloaded Metadata XML, then Save.



You should get following client details:



Step 8. Create UID attribute mapper

We used email SAML attribute in Nextcloud, so we have to create the same in Keycloak, on tab "Mappers->Create":


  • Name: email
  • Mapper Type: User Property
  • Property: email
  • SAML Attribute Name: email
  • SAML Attribute NameFormat: "URI Reference".

Step 9. Configure Logout (SLO Request)

SSO assumes that you can do "single logout" also, so Keycloak should be able to send logout message to the clients (aka Service Providers, aka Nextcloud).
Go to "Fine Grain SAML Endpoint Configuration" and specify attribute "Logout Service Redirect Binding URL" Nextcloud SAML URL, but ending with /sls:
http://172.20.140.180/nc/index.php/apps/user_saml/saml/sls




That's all. From this point you can do login to Nextcloud using Keycloak SAML protocol and successfully logout.

Comments are welcome and if I missed something, please, point this out.


Wordpress Parallax-One theme


We use quite cool Parallax-One theme for our cloud representation website. But recently I had the issue after changed website hostname.

All uploaded resources, like http://hostname/wp-content/uploads/2016/08/icon8.png, were linked to old hostname. The theme saved links in wordpress option "theme_mods_Parallax-One", and does not renew it after moving to new hostname.

First thought is quick fix by updating this option directly in database:

mysql -u root -p
mysql> use  wordpress;
mysql> select option_id, option_value from wp_options where option_name = 'theme_mods_Parallax-One'
mysql> | 169 | a:35:{i:0;b:0;s:26:"parallax_one_logos_content";
       s:1255:"[{"text":"undefined","link":"#",
       "image_url":"http:\/\/old_hostname\/wp-content\/uploads\/2016\/08\/turnkey_logo.jpg",
       "title":"undefined","subtitle": ...

But wp_options are PHP serialized objects, so if length of new_hostname and old_hostname is not equal, you have got a problem. You have to change length of the string encoded as follows:
s:<string length>:"string value"
And if there is missmatched something - theme will not load.

So let me introduce a little addition to Parallax-One theme. This patch adds input field 'Uploads Resource Hostname' to the inc/customizer.php. On saving changes it updates all upload URL hostnames to the specified hostname.

--- /root/customizer.php        2016-10-07 07:50:12.044940980 +0000
+++ customizer.php      2016-10-10 07:56:31.812266652 +0000
@@ -903,6 +903,36 @@
                'section'               => 'parallax_one_general_section',
                'priority'      => 12,
        ));
+
+        /** FIX change uploads resources hostname **/
+       $wp_customize->add_setting( 'parallax_one_uploads_hostname', array(
+               'sanitize_callback' => 'parallax_one_sanitize_input',
+                'validate_callback' => 'update_uploads_hostname',
+                'transport' => 'postMessage'
+       ));
+       $wp_customize->add_control( 'parallax_one_uploads_hostname', array(
+               'label'                 => esc_html__('Uploads Resource Hostname','parallax-one'),
+               'section'               => 'parallax_one_general_section',
+               'priority'      => 13,
+       ));
+        function update_uploads_hostname($validity, $value) {
+          if (!isset($value) || $value == "" || $value == $_SERVER['HOST_NAME']) return $validity;
+          $mods = get_option("theme_mods_Parallax-One");
+          $replace = function (&$url, $new_hostname) {
+            $url = preg_replace('/(http[s]?:(\\\\)?\/(\\\\)?\/)[^\\\\\/]+((\\\\)?\/wp-content(\\\\)?\/uploads)/', "$1$new_hostname$4", $url);
+            $url = preg_replace('/((http[s]?:)?(\\\\)?\/(\\\\)?\/)[^\\\\\/]+((\\\\)?\/wp-content(\\\\)?\/themes)/', "$1$new_hostname$5", $url);
+          };
+          foreach (array_keys($mods) as $key) {
+            if (is_string($mods[$key])) $replace($mods[$key], $value);
+            if (is_object($mods[$key])) { //for URLs in attachments, saved as objects, go through all string properties
+              foreach ($mods[$key] as $prop=>$pvalue) {
+                if (is_string($pvalue)) $replace($mods[$key]->$prop, $value);
+              }
+            }
+          }
+          update_option( "theme_mods_Parallax-One", $mods );
+          return $validity;
+        }


        /*********************************/

ManageIQ: Adding flexibility to Custom buttons

Custom buttons functionality allows you to hide/show buttons depending on user role, and assign buttons to particular service items. But sometimes buttons visibility could depend on different thing, like VM state or user quota etc.
In addition custom button are placed in one row at top bar, so it might become confusing if you have a lot of buttons for different purposes.
This workaround will help you to put your buttons wherever you want on a page, also allowing you to customize it with any picture or text you want.
First, you need to create haml view in any directory, for example  app/views/iba_custom/_service_custom_button.html
---
view ||= @record
= link_to_function(text, "$.post('/service/button/#{view.id}',{button_id: '#{button_id}', cls: 'Service', desc: '', id: '#{view.id}', pressed: 'custom_button'});", :style => "#{style}")

Then in service view you need to render this button with some params app/views/service/_svcs_show.html.haml
---
= render :partial => "iba_custom/service_custom_button", :locals => {:button_id => _("20"), :style => 'outline: none', :text => image_tag(image_path("image.png"), :class => "img", :width => 32, :height => 32)}

:button_id - id of custom button, which will be called from this button, button could be unassigned or not seen by any user, it still will be working
:style - any css style for your <a> tag which would appear in UI
:text - text of <a> tag. Could be plain text, image tag like in this case or any tag supported by rails haml formatting

Custom CFME reports: use VIEW

Hello everybody! Today we will create custom CFME reports using database view. It is interesting and not difficult thing.

 

Why

It's necessary to know all statistics about our enterprise. And now we have new stuff - Tenant. All in all, we use it to separate enterprise into departments. But standard reports can't give us possibilities to create statistics about every part of enterprise. So, our example - report about CPU, Memory and size on disks, which are belong to Tenant. 

How

1. Create read-only model InfoTenant:
---
class InfoTenant < ActiveRecord::Base
# Prevent creation of new records and modification to existing records  
  def readonly?
    return true
  end

# Prevent objects from being destroyed

  def before_destroy
  raise ActiveRecord::ReadOnlyRecord
  end
end

2. Create rails migration AddInfoTenants:
---
rails g migration AddInfoTenants
3. Change migration:
---
class AddInfoTenants < ActiveRecord::Migration
  def up
    execute '
      CREATE VIEW info_tenants AS 
      SELECT i.name AS name, SUM(t.cpu_total_cores) AS CPU, SUM((t.memory_mb/1024)) AS Memory_in_GB,
      SUM((q.size/1048576)) AS Size_on_dusk_GB
      FROM tenants AS i
      LEFT JOIN vms AS c ON i.id = c.tenant_id
      LEFT JOIN hardwares AS t ON c.id = t.vm_or_template_id
      LEFT JOIN disks AS q ON t.id = q.hardware_id
      GROUP BY i.name
     '
  end

  def down
    execute 'DROP VIEW info_tenants '
  end
end


4. Add new model into  enumerator @@base_tables of MiqExpression
---
@@base_tables = %w{
    Network
    AuditEvent
    AvailabilityZone
    BottleneckEvent
    Chargeback
    CloudResourceQuota
    CloudTenant
    Compliance
    ConfiguredSystemForeman
    ConfigurationManager
    EmsCloud
    EmsCluster
    EmsClusterPerformance
    EmsEvent
    EmsInfra
    ExtManagementSystem
    Flavor
    Host
    Vm
    ...
    InfoTenant

Result:

Good! Actually, you can use other tables of database and other combinations.

Sources:
https://habrahabr.ru/post/138949/
about CFME reports-1

VNC console: How to setup

Firstly, it's nessary to say, that VNC console is the best type of consoles, because you can connect to VMs on Vmware Vcenter and to VMs on RHEV. But it's easy to set up VMRC and you have a lot of problem with VNC. So, in this article you will read about configuration VNC, which was tested.


Configuration step by step

 


2. https://jamielinux.com/docs/openssl-certificate-authority/create-the-intermediate-pair.html

3. https://jamielinux.com/docs/openssl-certificate-authority/sign-server-and-client-certificates.html

4. Then you should create key without pass phrase(password). You can do it using this:
openssl rsa -in /path/to/originalkeywithpass.key.pem -out /path/to/newkeywithnopass.key.pem

5. You should set correct selinux context for files, which you will use:
chcon unconfined_u:object_r:httpd_config_t:s0 /path/to/certificate/file
chcon unconfined_u:object_r:httpd_config_t:s0 /path/to/chain/chainfile
chcon unconfined_u:object_r:httpd_config_t:s0 /path/to/key/keywithnopass

6. Set new settings for Apache in file /etc/httpd/conf.d/cfme-https-application.conf (or another file for earlier versions of ManageIQ EVM and CloudForms Management Engine  - read https://access.redhat.com/articles/449033)

SSLCertificateFile /path/to/certificate/file
SSLCertificateKeyFile /path/to/certificate/newkeywithnopass.key
SSLCertificateChainFile /path/to/certificate/chainfile

7. Copy newkeywithnopass.key and certificate in /var/www/miq/vmdb/certs/

8. Set new settings for websocket:

9. Restart appliance.

10. Import chain file in Authorities in browser.



Charts on dashboard in Manageiq Capablanca : scheme of generating content and fixing problem

 A lot of people, which use Manageiq Capablanca, faced with problem of charts on dashboard. In this post i will descibe you scheme of building charts and solution of problem with dashboard. Let's go!


Scheme of building content of charts



In generating of contents of charts all functions of model MiqWidget  are involved.
But the most interesting:
-   generate_content
- generate_report
- generate_report_result
In function generate_content, firstly function generate_report is calling and then generate_report_result is calling. So, we have content to build our charts. Then we go to MiqWidget::ChartContent
--
class MiqWidget::ChartContent < MiqWidget::ContentGeneration
  def generate(user_or_group)
    theme = user_or_group.settings.fetch_path(:display, :reporttheme) if user_or_group.kind_of?(User)

    # TODO: default reporttheme to MIQ since it doesn't look like we ever change it
    theme ||= "MIQ"
    report.to_chart(theme, false, MiqReport.graph_options)
    report.chart
  end
end

We can see, that all charts use standard graphic options(size, colours and other).
So, then we should to know more about function to_chart:

--
module MiqReport::Formatters::Graph
  extend ActiveSupport::Concern

  module ClassMethods
  ...
  def to_chart(theme = nil, show_title = false, graph_options = nil)
    ReportFormatter::ReportRenderer.render(Charting.format) do |e|
      e.options.mri           = self
      e.options.show_title    = show_title
      e.options.graph_options = graph_options unless graph_options.nil?
      e.options.theme         = theme
    end

  end
end
Charting.format = jqplot ( for all charts). But what about ReportFormatter::ReportRenderer.render?

--
module ReportFormatter
  class ReportRenderer < Ruport::Controller
    stage :document_header, :document_body, :document_footer
    finalize :document
    options { |o| o.mri = o.show_title = o.theme = o.table_width = o.alignment = o.graph_options = nil }
  end
end

How we can see, ReportFormatter::ReportRenderer use standard library ruport, which use module ChartCommon. And this module, finally, build contents of charts. Look at module ChartCommon and some important functions of it:
-
    def build_document_body
      return no_records_found_chart if mri.table.nil? || mri.table.data.blank?

      # find the highest chart value and set the units accordingly for large disk values (identified by GB in units)
      maxcols = 8
      divider = 1
      if graph_options[:units] == "GB" && !graph_options[:composite]
        maxval = 0
        mri.graph[:columns].each_with_index do |col, col_idx|
          next if col_idx >= maxcols
          newmax = mri.table.data.collect { |r| r[col].nil? ? 0 : r[col] }.sort.last
          maxval = newmax if newmax > maxval
        end
        if maxval > 10.gigabytes
          divider = 1.gigabyte
        elsif maxval > 10.megabytes
          graph_options[:units] = "MB"
          divider = 1.megabyte
        elsif maxval > 10.kilobytes
          graph_options[:units] = "KB"
          divider = 1.kilobyte
        else
          graph_options[:units] = "Bytes"
          graph_options[:decimals] = 0
          divider = 1
        end
        mri.title += " (#{graph_options[:units]})" unless graph_options[:units].blank?
      endfun = case graph_options[:chart_type]
            when :performance then :build_performance_chart # performance chart (time based)
            when :util_ts     then :build_util_ts_chart     # utilization timestamp chart (grouped columns)
            when :planning    then :build_planning_chart    # trend based planning chart
            else                                            # reporting charts
              mri.graph[:mode] == 'values' ? :build_reporting_chart_numeric : :build_reporting_chart
            end
      method(fun).call(maxcols, divider)
    end
    
    def build_reporting_chart_other
      save_key   = nil
      counter    = 0
      categories = []                      # Store categories and series counts in an array of arrays
      mri.table.data.each_with_index do |r, d_idx|
        if d_idx > 0 && save_key != r[mri.sortby[0]]
          save_key = nonblank_or_default(save_key)
          categories.push([save_key, counter])    # Push current category and count onto the array
          counter = 0
        end
        save_key = r[mri.sortby[0]]
        counter += 1
      end
      # add the last key/value to the categories and series arrays
      save_key = nonblank_or_default(save_key)
      categories.push([save_key, counter])        # Push last category and count onto the array

      categories.sort! { |a, b| b.last <=> a.last }
      (keep, show_other) = keep_and_show_other
      if keep < categories.length                      # keep the cathegories w/ highest counts
        other = categories.slice!(keep..-1)
        ocount = other.reduce(0) { |a, e| a + e.last } # sum up and add the other counts
        categories.push(["Other", ocount]) if show_other
      end

      series = categories.each_with_object(
        series_class.new(pie_type? ? :pie : :flat)) do |cat, a|
        a.push(:value => cat.last, :tooltip => "#{cat.first}: #{cat.last}")
      end

      # Pie charts put categories in legend, else in axis labels
      limit = pie_type? ? LEGEND_LENGTH : LABEL_LENGTH
      categories.collect! { |c| slice_legend(c[0], limit) }
      add_axis_category_text(categories)
      add_series(mri.headers[0], series)
    end 

Functions of module ChartCommon form content of charts belonging to MiqWidget, and this content is saved in database in table miq_widgets_contents.
Then, when we want to see at charts on dashboard this content is parsed by YAMl parser and converted in charts. Look at page of charts:

-
-#
  Parameters:
    widget -- MiqWidget instance
- width  ||= 350
- height ||= 250
.panel-body.chart_widget{:style => "text-align: center; padding: 0"}
  .mc{:id => "dd_w#{widget.id}_box",
    :style => @sb[:dashboards][@sb[:active_db]][:minimized].include?(widget.id) ? 'display:none' : ''}

    - if widget.contents_for_user(current_user).contents.blank?
      = _('No chart data found')
      \. . .
    - datum = widget.contents_for_user(current_user).contents
    - if Charting.data_ok?(datum)
      -# we need to count all charts to be able to display multiple
      -# charts on a dashboard screen

      - datum = widget.contents_for_user(current_user).contents
      - WidgetPresenter.chart_data.push(:xml => datum)
      - chart_index = WidgetPresenter.chart_data.length - 1
      - chart_data = YAML.load(datum) if Charting.backend == :jqplot

      = chart_local(chart_data,
                    :id     => "miq_widgetchart_#{chart_index}".html_safe,
                    :width  => width,
                    :height => height)
    - else
      = _('Invalid chart data. Try regenerating the widgets.')
 

Problem with charts and its solution

When you install new version of Manageiq - Capablanca, you can see this on dashboard:


The reason for this problem is that in ChartCommon data is generated
in the form of Hash, but we need YAML.
So, you should add one correction in MiqWidget::ChartContent:

-
class MiqWidget::ChartContent < MiqWidget::ContentGeneration
  def generate(user_or_group)
    theme = user_or_group.settings.fetch_path(:display, :reporttheme) if user_or_group.kind_of?(User)

    # TODO: default reporttheme to MIQ since it doesn't look like we ever change it
    theme ||= "MIQ"
    report.to_chart(theme, false, MiqReport.graph_options)
    report.chart.to_yaml
  end
end

 
Result:


CFME Reports: scheme and extensions

About scheme of Reports


It's necessary to say in this topic about model MiqExpression, which is stored information about tables, which are used for Reports. This model have four enumerators related with Reports:

1. Enumerator @@base_tables(not fully):
---
@@base_tables = %w{
    Network
    AuditEvent
    AvailabilityZone
    BottleneckEvent
    Chargeback
    CloudResourceQuota
    CloudTenant
    Compliance
    ConfiguredSystemForeman
    ConfigurationManager
    EmsCloud
    EmsCluster
    EmsClusterPerformance
    EmsEvent
    EmsInfra
    ExtManagementSystem
    Flavor
    Host
    Vm
    ...

This enumerator includes names of models, on which reports are built. We can see at snapshot:

2. Enumerator @@include_tables (not fully too):
---
@@include_tables = %w{
    advanced_settings
    audit_events
    availability_zones
    cloud_networks
    cloud_resource_quotas
    cloud_tenants
    compliances
    compliance_details
    configuration_profiles
    configuration_managers
    configured_systems
    customization_scripts
    customization_script_media
    customization_script_ptables
    disks
    networks
    hardware
    ...

This enumerator includes tables, which are not base for reports, but we can see and use these tables, because they are part of @@base_tables. For example, table networks belongs to table hardware, and this table belongs to table of model Vm. So, networks and hardware are included in @@include_tables, and we can see them in available tables for reports. See on snapshot:

 3. Enumerator EXCLUDE_COLUMNS (not fully too):
---
EXCLUDE_COLUMNS = %w{
    ^.*_id$
    ^id$
    ^min_derived_storage.*$
    ^max_derived_storage.*$
    assoc_ids
    capture_interval
    filters
    icon

    intervals_in_rollup

    max_cpu_ready_delta_summation
    max_cpu_system_delta_summation
    max_cpu_used_delta_summation
    max_cpu_wait_delta_summation
    max_derived_cpu_available
    max_derived_cpu_reserved
    max_derived_memory_available
    max_derived_memory_reserved

    memory_usage
    ...

This enumerator includes fields, which should not be displayed in reports. For example, we can't see field password in report, which is based on model EVM User,but  password stores in table of this model.

4. Enumerator EXCLUDE_EXCEPTIONS:
---
EXCLUDE_EXCEPTIONS = %w{
    capacity_profile_1_memory_per_vm_with_min_max
    capacity_profile_1_vcpu_per_vm_with_min_max
    capacity_profile_2_memory_per_vm_with_min_max
    capacity_profile_2_vcpu_per_vm_with_min_max
    chain_id
    guid
  }
 
This enumerator includes fields, which should not be displayed in reports too.

Our extension for Reports


Now, we know scheme of Reports and we can add custom extension.For example:
Our task: We can click on VM in report, which includes VM Name and IP adress, and we can go to this VM. Finally, IP adress should be displayed in base model for reports Perfomance - VMs.
Our actions:

---
class VmPerformance < MetricRollup
  default_scope { where "resource_type = 'VmOrTemplate' and resource_id IS NOT NULL" }

  belongs_to :host,        :foreign_key => :parent_host_id
  belongs_to :ems_cluster, :foreign_key => :parent_ems_cluster_id
  belongs_to :storage,     :foreign_key => :parent_storage_id
  belongs_to :vm,          :foreign_key => :resource_id, :class_name => 'VmOrTemplate'

  virtual_column :ipaddresses,    :type => :string_set

  def ipaddresses
    return self.vm.hardware.ipaddresses
  end

end
- new virtual column ipaddresses with type string_set;
 - definition of ipaddresses (we use vm - model VmPerformance belongs to model of this table).

 Results: