(HCL Connections) Admin Toolbox 2023

Ad04

Christoph Stoettner

Amsterdam, 2023-03-24

Christoph Stoettner

+49 173 8588719
stoeps@vegardit.com
stoeps.de
christophstoettner

stoeps@infosec.exchange

Senior Consultant at Vegard IT
- Linux (Slackware) since 1995
- IBM Domino since 1999
- IBM Connections since 2009
Experience in
- Migrations, Deployments
- Performance Analysis, Infrastructure
Focusing in
- Monitoring, Security
More and more
- DevOps stuff

Agenda

Give something back
Challenges with HCL Connections
Documentation (read and write)
Track website changes
Browser
- AddOns
- Proxies
- API

SSL / TLS
- testssl.sh, Keystore Explorer, OpenSSL
Configuration versioning and comparison
- Meld, Git, vimdiff
Linter
- xmllint, yamllint, config-lint
Kubernetes Tools
- kubens, kubectx, kubetail, k9s, kubeshark

Open Source Software

Give something back

Support Open Source developers!
- Tons of options, contribute, test, document, or recommend tools

If we want to see a lively open source scene, we need to keep actively using open source software, and not be afraid of trying out new ones. If we find something good, we shouldn’t hesitate to recommend it to others, so they can also benefit from it.

www.hongkiat.com/blog/open-source-community-give-back/

Challenges with HCL Connections

HCL Connections admin challenges

Huge backend options
- Database (DB2, Oracle MS SQL Server)
- Java backend (WebSphere Application Server)
- Container (Component Pack, PFKAP — Product Formerly Known as Pink)
  - Dependencies not documented
  - multiple places to search for log files
Many options to access Connections
- Browser, Plugins, Mobile App
Security
- Single Sign On
  - SPNEGO / Kerberos, SAML, LtpaToken
- SSL/TLS

The old IBM view

Often got this deployment plans in the first years
Lotus Wiki (official documentation)

Figure 1. Image IBM

More realistic

Almost reality

Documentation (read and write)

Official Documentation

Since version 8
Generated docs: opensource.hcltechsw.com/connections-doc/
Sources as Github project
- HCL-TECH-SOFTWARE/connections-doc
- Theoretically easier to send changes
Markdown format
MkDocs generated

Some issues with formatting, but useful to create internal documentation or convert to other formats like PDF, epub or docbook

Write Documentation

I write most documention in
pandoc converts from any of these formats to (e.g.)
- HTML
- PDF
- MS Word
Use corporate templates

txt based formats can be version controlled in git

Track changes in websites

Get informed of changes

RSS is not dead
- Some years we got all news over social media, but these are filtered
RSS helps a lot
- Tiny Tiny RSS, FreshRSS, RSS clients for your OS
HCL Documentation and Support Portal do not support RSS
Support Portal (ServiceNow) subscribe to mail updates
- new documents in a category, but all languages (duplicates)
- single documents (when logged in)
- update notification, but no indication what has changed
Documentation of Connections split into
- Documentation, Knowledge Base, Community Forum & Connections-automation project

changedetection.io

github.com/dgtlmoon/changedetection.io
- Apache-2.0 license
Docker images (docker-compose)
- runs on x86 or Raspberry Pi
Notification
- Mail
- RSS
- Teams, Google Chat, Gitter, Discord …
Filters to ignore changes like
- last changed 3 days ago (instead of showing the date)
- 102 views (counting views)

changedetection.io examples

Add link to page to monitor
Show diffs
Open page
RSS Link to all changes

Select only the important stuff

Copy xpath from developer tools inspector
or Visual Filter selector

Change date and view count

Filter & Triggers

Browser and Add Ons

Browser

DevTools (Firebug, Chrome Web Developer)
- HAR Analytics (HTTP Archive)
- Copy as curl
Addons
- SAML-tracer
- Multi-Account Container & Container Proxy
Intercept Proxy
- OWASP Zed Attack Proxy (ZAP), Burp Suite
- mitmproxy, Charles

Disable uBlock origin for Connections

Replace Fiddler with HAR

Webdeveloper Tools > Network, Enable persist logs, Reload the page
Right click > Save all as HAR
To analyze a recorded HAR, just drag&drop into your network tab

Copy `curl` command line

curl 'https://cnx7-rh8.stoeps.home/social/api/mwgraphql' -X POST \
    -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0' \
    -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' \
    -H 'Content-Type: application/json' -H 'authorization: Bearer 82ee99648d5327[...]b507e1' \
    -H 'Origin: https://cnx7-rh8.stoeps.home' -H 'Connection: keep-alive' \
    -H 'Referer: https://cnx7-rh8.stoeps.home/homepage/' \
    -H 'Cookie: JSESSIONID=0000wjHhwi-[...]1fvsegm22; ROLE_metrics-report-run=false; ROLE_admin=false; lang=en; \
        BAYEUX_BROWSER=ab06-15e90oty5xbqxl1mgaacexs6; ROLE_mail-user=true; blogsUser=Joe Jones2; \
        LtpaToken2=mBsFxSGe5j[...]B5fqZtEiRiA==' -H 'Sec-Fetch-Dest: empty' \
    -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' \
    --data-raw '{"query":"query {userprefs {applications {orient_me {defaultHomeLink}}}}","variables":{}}'

HAR and curl command include authorization tokens

SAML Tracer

Decrypt the SAML data
Check mappings (uid, mail addresses)

Multi Account Container

Very convenient to open ISC and Connections with different users
- ISC as user wasadmin, Connections as normal user
Test something with different users in one browser
- e.g. create content and check notification
- login with multiple accounts
Shares Cookies with containers of the same class
Private tab / window shares cookies of all private tabs
Container Proxy is an additional add-on
- set proxies for containers of one kind
- e.g. Burp Suite for one class, Tor for another

Container Proxy

Intercept proxies

Often used in Bug Bounty
Import HAR (ZAP)
See requests and responses
Bypass client side controls
Brute Force and Fuzz API or Logins

Figure 2. History

Figure 3. Site tree

OWASP ZAP

OWASP ZAP HUD

mitmproxy2swagger

Records flows
Script to convert flow to swagger
- github.com/alufers/mitmproxy2swagger
- Great to document unknown or undocumented APIs
- Can convert HAR to swagger too
Script generates a file with all found URL
- remove ignore and call script again
Run the command again
Import result to swagger editor editor.swagger.io/

mitmproxy2swagger -i flow -o swagger.yml -p https://preview.hclconnections.net

TLS and SSL

OpenSSL

Check TLS version 1.2 support

openssl s_client -connect stoeps.de:443 -servername stoeps.de -tls1_2

CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = Encryption Everywhere DV TLS CA - G1
verify return:1
depth=0 CN = *.stoeps.de
verify return:1
---
Certificate chain
 0 s:CN = *.stoeps.de

Check TLS version 1.3 support

openssl s_client -connect stoeps.de:443 -servername stoeps.de -tls1_3

CONNECTED(00000003)
406753CCBC7F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1584:SSL alert number 40
---
no peer certificate available

Check expiration date

openssl s_client -connect stoeps.de:443 -servername stoeps.de 2> /dev/null |  openssl x509 -noout  -dates

notBefore=Jul  9 00:00:00 2022 GMT
notAfter=Jul 24 23:59:59 2023 GMT

Check supported cipher suites

nmap
- nmap.org

nmap --script ssl-enum-ciphers -p 443 stoeps.de

Starting Nmap 7.80 ( https://nmap.org ) at 2023-04-16 16:22 CEST
Nmap scan report for stoeps.de (217.160.0.55)
Host is up (0.017s latency).
rDNS record for 217.160.0.55: 217-160-0-55.elastic-ssl.ui-r.com

PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 1.61 seconds

Check supported cipher suites

sslscan
- github.com/rbsec/sslscan

sslscan stoeps.de
...
Testing SSL server stoeps.de on port 443 using SNI name stoeps.de

  SSL/TLS Protocols:
SSLv2     disabled
SSLv3     disabled
TLSv1.0   disabled
TLSv1.1   disabled
TLSv1.2   enabled
TLSv1.3   disabled

...
  Supported Server Cipher(s):
Preferred TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA256       Curve 25519 DHE 253

`testssl.sh`

Does all of the scripts from the slides before testssl.sh
Install or run with docker / podman

podman run --rm -ti docker.io/drwetter/testssl.sh cnx8-ora.stoeps.home:443

###########################################################
    testssl.sh       3.2rc2 from https://testssl.sh/dev/

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-bad (1.0.2k-dev)" [~183 ciphers]
 on c36395b25607:/home/testssl/bin/openssl.Linux.x86_64
 (built: "Sep  1 14:03:44 2022", platform: "linux-x86_64")

 Start 2023-04-16 17:09:59                -->> 10.0.22.80:443 (cnx8-ora-was.stoeps.home) <<--

 A record via:           /etc/hosts
 rDNS (10.0.22.80):      --
 Service detected:       HTTP

`testssl.sh` — protocols and ciphers

Figure 4. OpenLDAP

Figure 5. nginx

`testssl.sh` — check application access

Figure 6. OpenLDAP

Figure 7. nginx

`testssl.sh` — check for vulnerabilities

Keystore Explorer — Import Signer for TDI

Create Keystore (JKS)
Examine > Examine SSL
- Add hostname & port
- Import

Keystore Explorer — Import Signer for TDI (2)

Save Keystore (asks for password)
Copy to your tdi solution directory
- Add to solution.properties

javax.net.ssl.trustStore=tdi-keystore.jks
{protect}-javax.net.ssl.trustStorePassword=password
javax.net.ssl.trustStoreType=jks

Enable SSL in profiles_tdi.properties

source_ldap_url=ldap://cnx-ds.stoeps.home:636
source_ldap_use_ssl=true

Versioning and Diff

Version Control - `git`

Versioning for these directories
- Dmgr01/config/cells/<cellname>/LotusConnections-config/
- <sharedDirectory>/customization
- tdisol
.gitignore
- *.jar
- *.xsd
Branches for new features
- Switching branches to test a feature, merge to keep
- Use on repository for test and production
Linux / Windows format conversion on the fly

Compare files

Compare files between production and testing
Side by side migration, compare directory trees
- I NEVER trust any documentation to find all changes
Software

Git to track changes

git can help a lot during migration (side-by-side)
Check if every change is commited in the "old" system
No need to copy files and folders
Commit new system to a branch
Compare on your desktop

git diff branch1 branch2

Diff with `git`

Can be used for versioning or diff

git diff --word-diff $f1 $f2

Can be used anywhere, no need to have a git repository

Meld

vimdiff

Installed with vim
Available on most server installations

vimdiff file1 file2

Figure 8. floatingoctothorpe.uk/2017/comparing-remote-files-with-vim.html

Linting

Linter

As standalone tools, or integrated into your favorite editor
- VIM, Emacs
- VS Code, notepad++
Examples
- Ansible-lint
  - Ansible: Linting playbooks, roles and collections
- yamllint
  - Kubernetes, Ansible
  - Autocomplete in editors, test in CI/CD pipeline
- config-lint
  - Validates: Terraform, Kubernetes, LintRules, YAML, JSON
- LanguageTool
  - Spellchecker, Grammar, Standalone, integrated in editor, browser add-on

Ansible Lint

ansible-lint playbooks
WARNING: PATH altered to include /usr/bin
WARNING  Listing 1 violation(s) that are fatal
syntax-check: couldn't resolve module/action 'xml'. This often indicates a misspelling,
missing collection, or incorrect module path.
roles/hcl/connections/clean_was_temp/tasks/main.yml:17:3 [WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
ERROR! couldn't resolve module/action 'xml'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-automation/roles/hcl/connections/clean_was_temp/tasks/main.yml': line 17, column 3,
but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name:                  Update versionStamp in LotusConnections-config.xml
  ^ here

Finished with 1 failure(s), 0 warning(s) on 71 files.

Ansible Lint

❯ ansible-lint -x yaml roles

[WARNING]: While constructing a mapping from /home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-
automation/roles/hcl/component-pack/tasks/setup_ingress.yml, line 45, column 3,
found a duplicate dict key (shell).
Using last defined value only.
[WARNING]: While constructing a mapping from <unicode string>, line 220, column 7,
found a duplicate dict key
(namespace). Using last defined value only.

fqcn-builtins: Use FQCN for builtin actions.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect

package-latest: Package installs should not use latest.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect

Finished with 2197 failure(s), 424 warning(s) on 628 files.

Yamllint

roles/third_party/ibm/db2-install/db2-restart/tasks/main.yml
  2:25      error    too many spaces after colon  (colons)
  3:25      error    too many spaces after colon  (colons)
  4:25      error    too many spaces after colon  (colons)
  9:25      warning  truthy value should be one of [false, true]  (truthy)
  9:81      error    line too long (158 > 80 characters)  (line-length)
  10:25     error    too many spaces after colon  (colons)
  11:25     error    too many spaces after colon  (colons)
  13:81     error    line too long (321 > 80 characters)  (line-length)
  15:4      error    wrong indentation: expected 4 but found 3  (indentation)

`xmllint`

Validates against xsd
Fast way to check files in LotusConnections-config
Not all config files do have a wsadmin option to checkout and validate

xmllint -noout -schema LotusConnections-config.xsd LotusConnections-config.xml

Check all xml in LCC

for i in $(ls *.xsd); do xmllint -noout -schema $i "${i%.xsd}.xml"; done
calendar-config.xml validates
communities-config.xml validates
communities-policy.xml validates
contentreview-config.xml validates
directory.services.xml validates
...

Kubernetes Tools

Kubernetes

Some tools to work faster on the console
- kubectx, kubens
- fzf for kubectl, kubectx
Get logs from running pods
- kubetail or stern
Speed up the most used tasks with kubectl
- K9s — (Win, Mac, Lnx)
Get some insights of the blackbox Kubernetes
- Linkerd, Istio

`kubectx`, `kubens`

Change context or namespace for kubectl
kubectl ns shows all namespaces
- select the namespace to set a new default
Speed up kubectl
- no need to type -n connections over and over again
kubectx or kubectl ctx
- set Kubernetes master and user to connect with kubectl
- useful if you administrate multiple Kubernetes clusters from one host

`kubetail`, `stern`

kubetail
- Display logs from multiple containers/pods
- Regular Expression or label to select
stern
- Shows running container, not all
- -c selects container

`k9s`

Replaces watch kubectl get pods for me
Check logs of pods and containers from the terminal ui
No complicated cli commands necessary

Istio

Componentpack is somehow a black box
No documentation on dependencies
- Which pods should I check when for example Orient Me isn’t working
- Which pods can you restart without affecting Customizer?
Istio
- Service Mesh
  - Traffic Management
  - Observability
  - Security

Istio — Installation

Fast, but unsecure

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.13.4 TARGET_ARCH=x86_64 sh -
cd istio-1.13.4
export PATH=$PWD/bin:$PATH
istioctl install --set profile=demo
istioctl manifest apply --set components.cni.enabled=true

kubectl get pods -n istio-system

NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-fgrhx                    1/1     Running   0          168m
istio-ingressgateway-76dcc86449-5z9rd   1/1     Running   0          168m
istiod-7664dfcb67-5wsgz                 1/1     Running   0          168m

Istio — Sidecar

Enable sidecars for namespace connections

kubectl label namespace connections istio-injection=enabled

Disable sidecar for Elasticsearch

kubectl edit statefulsets.apps/es-data-7
kubectl edit statefulsets.apps/es-master-7
kubectl edit deployment es-client-7

spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"

Restart all statefulsets and deployments

Install Kiali on top of Istio

Kiali is an observability console for Istio
- With service mesh configuration and validation capabilities
Helps you understand the structure and health of your service mesh

kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.13/samples/addons/kiali.yaml

kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.13/samples/addons/prometheus.yaml

Container

Work with local container
- Use tools
- Test
- Change images
Podman
Docker
Dive
- Analyze container layer and changes

Create videos

OBS

Open Broadcaster Software
- OBS Studio
Start creating videos for
- Speeding up your support cases
- Help your users
- Stream to any platform
Record single windows, or whole desktop
- add camera, audio
- logos

OBS Studio

Free Video Editors

Figure 9. OpenShot

Session Slides 2022

PDF version at engage.ug/engage2.nsf/Pages/sessionagenda2022
HTML version with embedded videos

(HCL Connections) Admin Toolbox 2023

Christoph Stoettner

Agenda

Open Source Software

Give something back

Challenges with HCL Connections

HCL Connections admin challenges

The old IBM view

More realistic

Almost reality

Documentation (read and write)

Official Documentation

Write Documentation

Track changes in websites

Get informed of changes

changedetection.io

changedetection.io examples

Select only the important stuff

Change date and view count

Filter & Triggers

Browser and Add Ons

Browser

Replace Fiddler with HAR

Copy curl command line

SAML Tracer

Multi Account Container

Container Proxy

Intercept proxies

OWASP ZAP

OWASP ZAP HUD

mitmproxy2swagger

TLS and SSL

OpenSSL

Check supported cipher suites

Check supported cipher suites

testssl.sh

testssl.sh — protocols and ciphers

testssl.sh — check application access

testssl.sh — check for vulnerabilities

Keystore Explorer — Import Signer for TDI

Keystore Explorer — Import Signer for TDI (2)

Versioning and Diff

Version Control - git

Compare files

Git to track changes

Diff with git

Meld

vimdiff

Linting

Linter

Ansible Lint

Ansible Lint

Yamllint

xmllint

Kubernetes Tools

Kubernetes

kubectx, kubens

kubetail, stern

k9s

Istio

Istio — Installation

Istio — Sidecar

Install Kiali on top of Istio

Container

Create videos

OBS

OBS Studio

Free Video Editors

Session Slides 2022

Copy `curl` command line

`testssl.sh`

`testssl.sh` — protocols and ciphers

`testssl.sh` — check application access

`testssl.sh` — check for vulnerabilities

Version Control - `git`

Diff with `git`

`xmllint`

`kubectx`, `kubens`

`kubetail`, `stern`

`k9s`