(HCL Connections) Admin Toolbox 2023


Christoph Stoettner

Amsterdam, 2023-03-24

Christoph Stoettner


+49 173 8588719

  • Senior Consultant at Vegard IT

    • Linux (Slackware) since 1995

    • IBM Domino since 1999

    • IBM Connections since 2009

  • Experience in

    • Migrations, Deployments

    • Performance Analysis, Infrastructure

  • Focusing in

    • Monitoring, Security

  • More and more

    • DevOps stuff


  • Give something back

  • Challenges with HCL Connections

  • Documentation (read and write)

  • Track website changes

  • Browser

    • AddOns

    • Proxies

    • API

  • SSL / TLS

    • testssl.sh, Keystore Explorer, OpenSSL

  • Configuration versioning and comparison

    • Meld, Git, vimdiff

  • Linter

    • xmllint, yamllint, config-lint

  • Kubernetes Tools

    • kubens, kubectx, kubetail, k9s, kubeshark

Open Source Software

Give something back

  • Support Open Source developers!

    • Tons of options, contribute, test, document, or recommend tools

If we want to see a lively open source scene, we need to keep actively using open source software, and not be afraid of trying out new ones. If we find something good, we shouldn’t hesitate to recommend it to others, so they can also benefit from it.

Challenges with HCL Connections

HCL Connections admin challenges

  • Huge backend options

    • Database (DB2, Oracle MS SQL Server)

    • Java backend (WebSphere Application Server)

    • Container (Component Pack, PFKAP — Product Formerly Known as Pink)

      • Dependencies not documented

      • multiple places to search for log files

  • Many options to access Connections

    • Browser, Plugins, Mobile App

  • Security

    • Single Sign On

      • SPNEGO / Kerberos, SAML, LtpaToken

    • SSL/TLS

The old IBM view

  • Often got this deployment plans in the first years

  • Lotus Wiki (official documentation)

netplan simple
con cluster
Figure 1. Image IBM

More realistic

connections netplan

Almost reality

connections netplan adv

Documentation (read and write)

Official Documentation

Some issues with formatting, but useful to create internal documentation or convert to other formats like PDF, epub or docbook

Write Documentation

  • I write most documention in

  • pandoc converts from any of these formats to (e.g.)

    • HTML

    • PDF

    • MS Word

  • Use corporate templates

txt based formats can be version controlled in git

Track changes in websites

Get informed of changes

  • RSS is not dead

    • Some years we got all news over social media, but these are filtered

  • RSS helps a lot

  • HCL Documentation and Support Portal do not support RSS

  • Support Portal (ServiceNow) subscribe to mail updates

    • new documents in a category, but all languages (duplicates)

    • single documents (when logged in)

    • update notification, but no indication what has changed

  • Documentation of Connections split into


  • github.com/dgtlmoon/changedetection.io

    • Apache-2.0 license

  • Docker images (docker-compose)

    • runs on x86 or Raspberry Pi

  • Notification

    • Mail

    • RSS

    • Teams, Google Chat, Gitter, Discord …​

  • Filters to ignore changes like

    • last changed 3 days ago (instead of showing the date)

    • 102 views (counting views)

changedetection.io examples

20230412 165221
  1. Add link to page to monitor

  2. Show diffs

  3. Open page

  4. RSS Link to all changes

Select only the important stuff

  • Copy xpath from developer tools inspector

  • or Visual Filter selector

20230412 182240

Change date and view count

20230412 183601

Filter & Triggers

20230412 182304

Browser and Add Ons


Disable uBlock origin for Connections

Replace Fiddler with HAR

  • Webdeveloper Tools > Network, Enable persist logs, Reload the page

  • Right click > Save all as HAR

  • To analyze a recorded HAR, just drag&drop into your network tab


Copy curl command line

curl 'https://cnx7-rh8.stoeps.home/social/api/mwgraphql' -X POST \
    -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0' \
    -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' \
    -H 'Content-Type: application/json' -H 'authorization: Bearer 82ee99648d5327[...]b507e1' \
    -H 'Origin: https://cnx7-rh8.stoeps.home' -H 'Connection: keep-alive' \
    -H 'Referer: https://cnx7-rh8.stoeps.home/homepage/' \
    -H 'Cookie: JSESSIONID=0000wjHhwi-[...]1fvsegm22; ROLE_metrics-report-run=false; ROLE_admin=false; lang=en; \
        BAYEUX_BROWSER=ab06-15e90oty5xbqxl1mgaacexs6; ROLE_mail-user=true; blogsUser=Joe Jones2; \
        LtpaToken2=mBsFxSGe5j[...]B5fqZtEiRiA==' -H 'Sec-Fetch-Dest: empty' \
    -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' \
    --data-raw '{"query":"query {userprefs {applications {orient_me {defaultHomeLink}}}}","variables":{}}'
HAR and curl command include authorization tokens

SAML Tracer

  • Decrypt the SAML data

  • Check mappings (uid, mail addresses)

saml tracer attribute
20230415 141926
saml tracer

Multi Account Container

  • Very convenient to open ISC and Connections with different users

    • ISC as user wasadmin, Connections as normal user

  • Test something with different users in one browser

    • e.g. create content and check notification

    • login with multiple accounts

  • Shares Cookies with containers of the same class

  • Private tab / window shares cookies of all private tabs

  • Container Proxy is an additional add-on

    • set proxies for containers of one kind

    • e.g. Burp Suite for one class, Tor for another

Container Proxy


Intercept proxies

  • Often used in Bug Bounty

  • Import HAR (ZAP)

  • See requests and responses

  • Bypass client side controls

  • Brute Force and Fuzz API or Logins

zap history
Figure 2. History
zap tree
Figure 3. Site tree


zap wide


zap hud3


  • Records flows

  • Script to convert flow to swagger

  • Script generates a file with all found URL

    • remove ignore and call script again

  • Run the command again

  • Import result to swagger editor editor.swagger.io/

mitmproxy2swagger -i flow -o swagger.yml -p https://preview.hclconnections.net



Check TLS version 1.2 support
openssl s_client -connect stoeps.de:443 -servername stoeps.de -tls1_2

depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = Encryption Everywhere DV TLS CA - G1
verify return:1
depth=0 CN = *.stoeps.de
verify return:1
Certificate chain
 0 s:CN = *.stoeps.de
Check TLS version 1.3 support
openssl s_client -connect stoeps.de:443 -servername stoeps.de -tls1_3

406753CCBC7F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1584:SSL alert number 40
no peer certificate available
Check expiration date
openssl s_client -connect stoeps.de:443 -servername stoeps.de 2> /dev/null |  openssl x509 -noout  -dates

notBefore=Jul  9 00:00:00 2022 GMT
notAfter=Jul 24 23:59:59 2023 GMT

Check supported cipher suites

nmap --script ssl-enum-ciphers -p 443 stoeps.de
Starting Nmap 7.80 ( https://nmap.org ) at 2023-04-16 16:22 CEST
Nmap scan report for stoeps.de (
Host is up (0.017s latency).
rDNS record for 217-160-0-55.elastic-ssl.ui-r.com

443/tcp open  https
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 1.61 seconds

Check supported cipher suites

sslscan stoeps.de
Testing SSL server stoeps.de on port 443 using SNI name stoeps.de

  SSL/TLS Protocols:
SSLv2     disabled
SSLv3     disabled
TLSv1.0   disabled
TLSv1.1   disabled
TLSv1.2   enabled
TLSv1.3   disabled

  Supported Server Cipher(s):
Preferred TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA256       Curve 25519 DHE 253


  • Does all of the scripts from the slides before testssl.sh

  • Install or run with docker / podman

podman run --rm -ti docker.io/drwetter/testssl.sh cnx8-ora.stoeps.home:443
    testssl.sh       3.2rc2 from https://testssl.sh/dev/

      This program is free software. Distribution and
             modification under GPLv2 permitted.

       Please file bugs @ https://testssl.sh/bugs/


 Using "OpenSSL 1.0.2-bad (1.0.2k-dev)" [~183 ciphers]
 on c36395b25607:/home/testssl/bin/openssl.Linux.x86_64
 (built: "Sep  1 14:03:44 2022", platform: "linux-x86_64")

 Start 2023-04-16 17:09:59                -->> (cnx8-ora-was.stoeps.home) <<--

 A record via:           /etc/hosts
 rDNS (      --
 Service detected:       HTTP

testssl.sh — protocols and ciphers

Figure 4. OpenLDAP
Figure 5. nginx

testssl.sh — check application access

Figure 6. OpenLDAP
Figure 7. nginx

testssl.sh — check for vulnerabilities


Keystore Explorer — Import Signer for TDI

  • Create Keystore (JKS)

  • Examine > Examine SSL

    • Add hostname & port

    • Import

kse tdi1

Keystore Explorer — Import Signer for TDI (2)

  • Save Keystore (asks for password)

  • Copy to your tdi solution directory

    • Add to solution.properties

  • Enable SSL in profiles_tdi.properties

kse tdi3

Versioning and Diff

Version Control - git

  • Versioning for these directories

    • Dmgr01/config/cells/<cellname>/LotusConnections-config/

    • <sharedDirectory>/customization

    • tdisol

  • .gitignore

    • *.jar

    • *.xsd

  • Branches for new features

    • Switching branches to test a feature, merge to keep

    • Use on repository for test and production

  • Linux / Windows format conversion on the fly

Compare files

Git to track changes

  • git can help a lot during migration (side-by-side)

  • Check if every change is commited in the "old" system

  • No need to copy files and folders

  • Commit new system to a branch

  • Compare on your desktop

git diff branch1 branch2

Diff with git

  • Can be used for versioning or diff

git diff --word-diff $f1 $f2
20230417 165109
  • Can be used anywhere, no need to have a git repository




  • Installed with vim

  • Available on most server installations

vimdiff file1 file2
vimdiff with a remote file



  • As standalone tools, or integrated into your favorite editor

    • VIM, Emacs

    • VS Code, notepad++

  • Examples

    • Ansible-lint

      • Ansible: Linting playbooks, roles and collections

    • yamllint

      • Kubernetes, Ansible

      • Autocomplete in editors, test in CI/CD pipeline

    • config-lint

      • Validates: Terraform, Kubernetes, LintRules, YAML, JSON

    • LanguageTool

      • Spellchecker, Grammar, Standalone, integrated in editor, browser add-on

Ansible Lint

ansible-lint playbooks
WARNING: PATH altered to include /usr/bin
WARNING  Listing 1 violation(s) that are fatal
syntax-check: couldn't resolve module/action 'xml'. This often indicates a misspelling,
missing collection, or incorrect module path.
roles/hcl/connections/clean_was_temp/tasks/main.yml:17:3 [WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
ERROR! couldn't resolve module/action 'xml'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-automation/roles/hcl/connections/clean_was_temp/tasks/main.yml': line 17, column 3,
but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name:                  Update versionStamp in LotusConnections-config.xml
  ^ here

Finished with 1 failure(s), 0 warning(s) on 71 files.

Ansible Lint

❯ ansible-lint -x yaml roles

[WARNING]: While constructing a mapping from /home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-
automation/roles/hcl/component-pack/tasks/setup_ingress.yml, line 45, column 3,
found a duplicate dict key (shell).
Using last defined value only.
[WARNING]: While constructing a mapping from <unicode string>, line 220, column 7,
found a duplicate dict key
(namespace). Using last defined value only.

fqcn-builtins: Use FQCN for builtin actions.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect

package-latest: Package installs should not use latest.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect

Finished with 2197 failure(s), 424 warning(s) on 628 files.


  2:25      error    too many spaces after colon  (colons)
  3:25      error    too many spaces after colon  (colons)
  4:25      error    too many spaces after colon  (colons)
  9:25      warning  truthy value should be one of [false, true]  (truthy)
  9:81      error    line too long (158 > 80 characters)  (line-length)
  10:25     error    too many spaces after colon  (colons)
  11:25     error    too many spaces after colon  (colons)
  13:81     error    line too long (321 > 80 characters)  (line-length)
  15:4      error    wrong indentation: expected 4 but found 3  (indentation)


  • Validates against xsd

  • Fast way to check files in LotusConnections-config

  • Not all config files do have a wsadmin option to checkout and validate

xmllint -noout -schema LotusConnections-config.xsd LotusConnections-config.xml
Check all xml in LCC
for i in $(ls *.xsd); do xmllint -noout -schema $i "${i%.xsd}.xml"; done
calendar-config.xml validates
communities-config.xml validates
communities-policy.xml validates
contentreview-config.xml validates
directory.services.xml validates

Kubernetes Tools


  • Some tools to work faster on the console

  • Get logs from running pods

  • Speed up the most used tasks with kubectl

    • K9s — (Win, Mac, Lnx)

  • Get some insights of the blackbox Kubernetes

kubectx, kubens

  • Change context or namespace for kubectl

  • kubectl ns shows all namespaces

    • select the namespace to set a new default

  • Speed up kubectl

    • no need to type -n connections over and over again

  • kubectx or kubectl ctx

    • set Kubernetes master and user to connect with kubectl

    • useful if you administrate multiple Kubernetes clusters from one host

kubectx interactive

kubetail, stern

  • kubetail

    • Display logs from multiple containers/pods

    • Regular Expression or label to select

  • stern

    • Shows running container, not all

    • -c selects container



  • Replaces watch kubectl get pods for me

  • Check logs of pods and containers from the terminal ui

  • No complicated cli commands necessary


  • Componentpack is somehow a black box

  • No documentation on dependencies

    • Which pods should I check when for example Orient Me isn’t working

    • Which pods can you restart without affecting Customizer?

  • Istio

    • Service Mesh

      • Traffic Management

      • Observability

      • Security

Istio — Installation

Fast, but unsecure
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.13.4 TARGET_ARCH=x86_64 sh -
cd istio-1.13.4
export PATH=$PWD/bin:$PATH
istioctl install --set profile=demo
istioctl manifest apply --set components.cni.enabled=true

kubectl get pods -n istio-system

NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-fgrhx                    1/1     Running   0          168m
istio-ingressgateway-76dcc86449-5z9rd   1/1     Running   0          168m
istiod-7664dfcb67-5wsgz                 1/1     Running   0          168m

Istio — Sidecar

Enable sidecars for namespace connections
kubectl label namespace connections istio-injection=enabled
Disable sidecar for Elasticsearch
kubectl edit statefulsets.apps/es-data-7
kubectl edit statefulsets.apps/es-master-7
kubectl edit deployment es-client-7
        sidecar.istio.io/inject: "false"
  • Restart all statefulsets and deployments

Install Kiali on top of Istio

  • Kiali is an observability console for Istio

    • With service mesh configuration and validation capabilities

  • Helps you understand the structure and health of your service mesh

kubectl apply -f \

kubectl apply -f \


  • Work with local container

    • Use tools

    • Test

    • Change images

  • Podman

  • Docker

  • Dive

    • Analyze container layer and changes

Create videos


  • Open Broadcaster Software

    • OBS Studio

  • Start creating videos for

    • Speeding up your support cases

    • Help your users

    • Stream to any platform

  • Record single windows, or whole desktop

    • add camera, audio

    • logos

OBS Studio

20230420 212111

Free Video Editors

20230420 213006
Figure 9. OpenShot

Session Slides 2022