(HCL Connections) Admin Toolbox

Ad04

Christoph Stoettner

@stoeps

Bruges, 24.05.2022

Christoph Stoettner

  • Senior Consultant at Vegard IT

    • Linux (Slackware) since 1995

    • IBM Domino since 1999

    • IBM Connections since 2009

  • Experience in

    • Migrations, Deployments

    • Performance Analysis, Infrastructure

  • Focusing in

    • Monitoring, Security

  • More and more

    • DevOps stuff

Open Source software - Give something back

  • Support Open Source developers!

    • Tons of options, contribute, test, document, or recommend tools

If we want to see a lively open source scene, we need to keep actively using open source software, and not be afraid of trying out new ones. If we find something good, we shouldn’t hesitate to recommend it to others, so they can also benefit from it.

https://www.hongkiat.com/blog/open-source-community-give-back/

HCL Connections admin challenges

  • Huge backend options

    • Database (DB2, Oracle MS SQL)

    • Java backend (WebSphere Application Server)

    • Container (Component Pack, PFKAP — Product Formerly Known as Pink)

      • Dependencies not documented

      • multiple places to search for log files

  • Multiple options to access Connections

    • Browser, Plugins, Mobile App

  • Security

    • Single Sign On

      • SPNEGO / Kerberos, SAML, LtpaToken

    • SSL / TLS

The old IBM view

  • Often got this deployment plans in the first years

  • Lotus Wiki (official documentation)

netplan simple
con cluster
Figure 1. Image IBM

More realistic

connections netplan

And now mostly meeting reality

connections netplan adv

Check after installation and updates

  • Is the environment working as expected

    • Smoke testing

  • Does it scale for your planned user count

    • Load testing

pexels rafael guajardo 604684

Tools for Testing

K6 example

import encoding from 'k6/encoding';
import http from 'k6/http';
import { check } from 'k6';
const username = 'jjones3';
const password = 'password';

export const options = {
    insecureSkipTLSVerify: true
};
export default function () {
  const credentials = `${username}:${password}`;
  const url = `https://${credentials}@cnx7-rh8.stoeps.home/wikis/home`;
  let res = http.get(url);
  check(res, {
    'status is 200': (r) => r.status === 200,
    'LtpaToken2': (r) => r.headers.LtpaToken2 !== 0,
  });
}

K6 result - single user connect

k6 result1

k6 - GraphQL load testing

k6 without error
k6 with error

Selenium Example - Python

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService

options = webdriver.ChromeOptions()
options.set_capability('acceptSslCerts', True)
service = ChromeService(executable_path='/snap/chromium/1985/usr/lib/chromium-browser/chromedriver')
driver = webdriver.Chrome(service=service, options=options)

driver.maximize_window()
driver.get("https://cnx7-rh8.stoeps.home/wikis/login")
driver.find_element(by=By.ID, value="username").send_keys('jjones3')
driver.find_element(by=By.ID, value="password").send_keys("password")
driver.find_element(by=By.CLASS_NAME, value="lotusBtnSpecial").click()

Logs

Browser

Disable uBlock origin for Connections

Replace Fiddler with HAR

  • Webdeveloper Tools > Network, Enable persist logs, Reload the page

  • Right click > Save all as HAR

  • To analyze a recorded HAR, just drag&drop into your network tab

saveashar

Copy curl command line from WebDev Tools

copyascurl
curl 'https://cnx7-rh8.stoeps.home/social/api/mwgraphql' -X POST \
    -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0' \
    -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' \
    -H 'Content-Type: application/json' -H 'authorization: Bearer 82ee99648d5327[...]b507e1' \
    -H 'Origin: https://cnx7-rh8.stoeps.home' -H 'Connection: keep-alive' \
    -H 'Referer: https://cnx7-rh8.stoeps.home/homepage/' \
    -H 'Cookie: JSESSIONID=0000wjHhwi-[...]1fvsegm22; ROLE_metrics-report-run=false; ROLE_admin=false; lang=en; BAYEUX_BROWSER=ab06-15e90oty5xbqxl1mgaacexs6; ROLE_mail-user=true; blogsUser=Joe Jones2; LtpaToken2=mBsFxSGe5j[...]B5fqZtEiRiA==' -H 'Sec-Fetch-Dest: empty' \
    -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' \
    --data-raw '{"query":"query {userprefs {applications {orient_me {defaultHomeLink}}}}","variables":{}}'
HAR and curl command include authorization tokens

SAML Tracer

  • Decrypt the SAML data

  • Check mappings (uid, mail addresses)

saml tracer attribute
saml tracer

Multi Account Container

  • Very convenient to open ISC and Connections with different users

    • ISC as user wasadmin, Connections as normal user

  • Test something with different users in one browser

    • e.g. create content and check notification

    • login with multiple accounts

  • Shares Cookies with containers of the same class

  • Private tab / window shares cookies of all private tabs

  • Container Proxy is an additional add-on

    • set proxies for containers of one kind

    • e.g. Burp Suite for one class, Tor for another

Multi Account Container — Container Proxy

containeraddon
containerproxy

Intercept proxies

  • Often used in Bug Bounty

  • Import HAR (ZAP)

  • See requests and responses

  • Bypass client side controls

  • Brute Force and Fuzz API or Logins

zap history
Figure 2. History
zap tree
Figure 3. Site tree

Intercept proxies example - OWASP ZAP

zap wide

Intercept proxies example - OWASP ZAP HUD

zap hud3

SSL

testssl.sh — protocols and ciphers

testssl1
Figure 4. OpenLDAP
testssl4
Figure 5. nginx

testssl.sh — check application access

testssl3
Figure 6. OpenLDAP
testssl5
Figure 7. nginx

testssl.sh — check for vulnerabilities

testssl2

Keystore Explorer — Import Signer for TDI

  • Create Keystore (JKS)

  • Examine > Examine SSL

    • Add hostname & port

    • Import

kse tdi1
kse1

Keystore Explorer — Import Signer for TDI (2)

  • Save Keystore (asks for password)

  • Copy to your tdi solution directory

    • Add to solution.properties

javax.net.ssl.trustStore=tdi-keystore.jks
{protect}-javax.net.ssl.trustStorePassword=password
javax.net.ssl.trustStoreType=jks

  • Enable SSL in profiles_tdi.properties

source_ldap_url=ldap://cnx-ds.stoeps.home:636
source_ldap_use_ssl=true
kse2
kse tdi3

Version Control

  • git

  • Versioning for these directories

    • Dmgr01/config/cells/<cellname>/LotusConnections-config/

    • <sharedDirectory>/customization

    • tdisol

  • .gitignore

    • *.jar

    • *.xsd

  • Branches for new features

    • Switching branches to test a feature, merge to keep

  • Linux / Windows format conversion on the fly

Compare files

  • Compare files between production and testing

  • Side by side migration, compare directory trees

    • NEVER trust your documentation to find all changes

  • Software

Meld

meld1

Linter

  • As standalone tools, or integrated into your favorite editor

    • VIM, Emacs

    • VS Code, notepad++

  • Examples

    • Ansible-lint

      • Ansible: Linting playbooks, roles and collections

    • yamllint

      • Kubernetes, Ansible

      • Autocomplete in editors, test in CI/CD pipeline

    • config-lint

      • Validates: Terraform, Kubernetes, LintRules, YAML, JSON

    • LanguageTool

      • Spellchecker, Grammar, Standalone, integrated in editor, browser add-on

Ansible Lint

ansible-lint playbooks
WARNING: PATH altered to include /usr/bin
WARNING  Listing 1 violation(s) that are fatal
syntax-check: couldn't resolve module/action 'xml'. This often indicates a misspelling,
missing collection, or incorrect module path.
roles/hcl/connections/clean_was_temp/tasks/main.yml:17:3 [WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
ERROR! couldn't resolve module/action 'xml'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-automation/roles/hcl/connections/clean_was_temp/tasks/main.yml': line 17, column 3,
but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name:                  Update versionStamp in LotusConnections-config.xml
  ^ here

Finished with 1 failure(s), 0 warning(s) on 71 files.

Ansible Lint

❯ ansible-lint -x yaml roles

[WARNING]: While constructing a mapping from /home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-
automation/roles/hcl/component-pack/tasks/setup_ingress.yml, line 45, column 3,
found a duplicate dict key (shell).
Using last defined value only.
[WARNING]: While constructing a mapping from <unicode string>, line 220, column 7,
found a duplicate dict key
(namespace). Using last defined value only.

fqcn-builtins: Use FQCN for builtin actions.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect

package-latest: Package installs should not use latest.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect

Finished with 2197 failure(s), 424 warning(s) on 628 files.

Yamllint

roles/third_party/ibm/db2-install/db2-restart/tasks/main.yml
  2:25      error    too many spaces after colon  (colons)
  3:25      error    too many spaces after colon  (colons)
  4:25      error    too many spaces after colon  (colons)
  9:25      warning  truthy value should be one of [false, true]  (truthy)
  9:81      error    line too long (158 > 80 characters)  (line-length)
  10:25     error    too many spaces after colon  (colons)
  11:25     error    too many spaces after colon  (colons)
  13:81     error    line too long (321 > 80 characters)  (line-length)
  15:4      error    wrong indentation: expected 4 but found 3  (indentation)

Extract fixes and jars

  • 7-zip

  • Check fixes jar for version and replaced files/paths

  • Extract jars (Java bundles, CFixes)

    • get customization files, check for changes

    • remove classes (Log4Shell workaround)

Decompiler (find issues)

  • jd-gui

    • Java Decompiler

  • Ghidra

    • A software reverse engineering (SRE) suite of tools developed by NSA

Icon java 64
ghidra

Check product version of a fix

  • Extract jar-file and Search efixDriver file

❯ cat KB0095928.efixDriver
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE efix-driver SYSTEM "applied.dtd">
<efix-driver
	id="KB0095928"
	short-description="Mobile: APNS Certificate for 2022"
	long-description="Mobile: APNS Certificate for 2022"
	build-date="01/07/2022"
	build-version="20220107.1054">

	<apar-info
		number="KB0095928"
		date="01/07/2022"
		short-description="Mobile: APNS Certificate for 2022"
		long-description="Mobile: APNS Certificate for 2022"
		/>

	<product-prereq
		product-id="mobile"
		build-version="6.0.0.0_CR6"  (1)
		build-date="*"
		build-level="*"/>
1Check build version

jd-gui

  • Example check Mobile APNS update file

jd gui1

  • Check ear-files, find trace settings

jd gui2

Ghidra

  • Decompiler

  • Check binaries for configuration strings

  • Example shows mod_ibm_upload.so

ghidra mod upload

Database (JDBC) Client — DBeaver Community

  • Supports DB2 and Elasticsearch

dbeaver4
dbeaver5

  • Converts binary ids (Files, Wikis) to UUID format for better readability

dbeaver1
dbeaver2
dbeaver3

Security

  • trivy

    • File System

    • Registry (Container)

  • Popeye

    • Scans live Kubernetes cluster

    • Detects misconfigurations

    • Helps you to ensure that best practices are in place

Trivy — container images

trivy image --input ~/vmware/software/cp_7.0.0.2/hybridcloud/images/admin-portal.tar -s CRITICAL

/home/stoeps/vmware/software/cp_7.0.0.2/hybridcloud/images/admin-portal.tar (alpine 3.12.0)
===========================================================================================
Total: 8 (CRITICAL: 8)

+----------+------------------+----------+-------------------+---------------+---------------------------------------+
|  LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |                 TITLE                 |
+----------+------------------+----------+-------------------+---------------+---------------------------------------+
|apk-tools | CVE-2021-36159   | CRITICAL | 2.10.5-r1         | 2.10.7-r0     | libfetch before 2021-07-26, as        |
|          |                  |          |                   |               | used in apk-tools, xbps, and          |
|          |                  |          |                   |               | other products, mishandles...         |
|          |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-36159 |
+----------+------------------+          +-------------------+---------------+---------------------------------------+
..
+----------+------------------+          +-------------------+---------------+---------------------------------------+
|ssl_client| CVE-2022-28391   |          | 1.31.1-r16        | 1.31.1-r22    | busybox: remote attackers may execute |
|          |                  |          |                   |               | arbitrary code if netstat is used     |
|          |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2022-28391 |
+----------+------------------+----------+-------------------+---------------+---------------------------------------+

Trivy — Filesystem

[root@cnx7-rh8-was ConnectionsCell]# trivy rootfs --ignore-unfixed -s CRITICAL Dogear.ear
[[C2022-05-18T10:18:07.039Z	INFO	Number of language-specific files: 1
2022-05-18T10:18:07.039Z	INFO	Detecting jar vulnerabilities...
2022-05-18T10:18:07.043Z	INFO	Table result includes only package filenames. Use '--format json' option to get the full path to the package file.

Java (jar)
Total: 4 (CRITICAL: 4)
┌───────────────────────────────────────────┬──────────────────┬──────────┬────────────┬────────────┬──────────────────────────────────────────────────────────────┐
│                           Library         │  Vulnerability   │ Severity │ Installed V│ Fixed Versi│                            Title                             │
├───────────────────────────────────────────┼──────────────────┼──────────┼────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ com.googlecode.owasp-java-html-sanitizer: │ CVE-2021-42575   │ CRITICAL │ 20171016.1 │ 20211018.1 │ owasp-java-html-sanitizer: improper policies enforcement may │
│ owasp-java-html-sa-nitizer (owasp-java-   │                  │          │            │            │ lead to remote code execution                                │
│ html-sanitizer-20171016.1.jar)            │                  │          │            │            │ https://avd.aquasec.com/nvd/cve-2021-42575                   │
├───────────────────────────────────────────┼──────────────────┤          ├────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ commons-collections:commons-collections   │ CVE-2015-7501    │          │ 3.2.1      │ 3.2.2      │ apache-commons-collections: InvokerTransformer code          │
│ (commons-collections-3.2.1.jar)           │                  │          │            │            │ execution during deserialisation                             │
│                                           │                  │          │            │            │ https://avd.aquasec.com/nvd/cve-2015-7501                    │
├───────────────────────────────────────────┼──────────────────┼──────────┼────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ commons-fileupload:commons-fileupload     │ CVE-2016-1000031 │ CRITICAL │ 1.2.1      │ 1.3.3      │ Apache Commons FileUpload: DiskFileItem file manipulation    │
│ (commons-fileupload-1.2.1.jar)            │                  │          │            │            │ https://avd.aquasec.com/nvd/cve-2016-1000031                 │
└───────────────────────────────────────────┴──────────────────┴──────────┴────────────┴────────────┴──────────────────────────────────────────────────────────────┘

popeye

popeye

Kubernetes

  • Some tools to work faster on the console

  • Get logs from running pods

  • Speed up the most used tasks with kubectl

    • K9s — (Win, Mac, Lnx)

  • Get some insights of the blackbox Kubernetes

kubectx, kubens

  • Change context or namespace for kubectl

  • kubectl ns shows all namespaces

    • select the namespace to set a new default

  • Speed up kubectl

    • no need to type -n connections over and over again

  • kubectx or kubectl ctx

    • set Kubernetes master and user to connect with kubectl

    • useful if you administrate multiple Kubernetes clusters from one host

kubens
kubectx interactive
kubectx

kubetail, stern

  • kubetail

    • Display logs from multiple containers/pods

    • Regular Expression or label to select

  • stern

    • Shows running container, not all

    • -c selects container

kubetail
stern

k9s

  • Replaces watch kubectl get pods for me

  • Check logs of pods and containers from the terminal ui

  • No complicated cli commands necessary

Istio

  • Componentpack is somehow a black box

  • No documentation on dependencies

    • Which pods should I check when for example Orient Me isn’t working

    • Which pods can you restart without affecting Customizer?

  • Istio

    • Service Mesh

      • Traffic Management

      • Observability

      • Security

Istio checking ComponentPack — Installation

Fast, but unsecure
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.13.4 TARGET_ARCH=x86_64 sh -
cd istio-1.13.4
export PATH=$PWD/bin:$PATH
istioctl install --set profile=demo
istioctl manifest apply --set components.cni.enabled=true

kubectl get pods -n istio-system

NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-fgrhx                    1/1     Running   0          168m
istio-ingressgateway-76dcc86449-5z9rd   1/1     Running   0          168m
istiod-7664dfcb67-5wsgz                 1/1     Running   0          168m

Istio checking ComponentPack — Sidecar

Enable sidecars for namespace connections
kubectl label namespace connections istio-injection=enabled
Disable sidecar for Elasticsearch
kubectl edit statefulsets.apps/es-data-7
kubectl edit statefulsets.apps/es-master-7
kubectl edit deployment es-client-7
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"

  • Restart all statefulsets and deployments

Install Kiali on top of Istio

  • Kiali is an observability console for Istio

    • With service mesh configuration and validation capabilities

  • Helps you understand the structure and health of your service mesh

kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.13/samples/addons/kiali.yaml

kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.13/samples/addons/prometheus.yaml

Documentation

  • I write most documention in

  • pandoc converts from any of these formats to (e.g.)

    • HTML

    • PDF

    • MS Word

    • …​

  • txt based formats can be version controlled in git

Container

  • Work with local container

    • Use tools

    • Test

    • Change images

  • Podman

  • Docker

  • Dive

    • Analyze container layer and changes

Some more useful tools for daily work

Session Slides

sessionslides

Links for all tools

Links for all tools (2)

Links for all tools (3)

Links for all tools (4)

Links for all tools (5)