Troubleshooting (Enterprise) Web Applikationen mit Open Source Tools

Christoph Stoettner

Bonn-St. Augustin, 21.08.2022

Christoph Stoettner (stoeps)

  • mastodon brands stoeps@chaos.social —  @stoeps — blog solid stoeps.de

  • Experience in "Enterprise" Application Server

    • Migrations, Deployments

    • Performance Analysis, Infrastructure

  • Focusing in

    • Monitoring, Security

    • Troubleshooting

  • More and more

    • DevOps, Cloud, Automation

Open Source software - Give something back

  • Support Open Source developers!

    • Tons of options, contribute, test, document, or recommend tools

If we want to see a lively open source scene, we need to keep actively using open source software, and not be afraid of trying out new ones.

If we find something good, we shouldn’t hesitate to recommend it to others, so they can also benefit from it.

— Anna Monus

Kann man auch alles mit Excel machen, aber…​

List of Tools

  • Complete collection: https://stoeps.de/tiu

    • Short description

    • License

  • Most of the tools shown in this presentation

    • Are open source

    • Available for Linux, Mac OS and Windows

tiu

Some years ago

  • Just a web server

  • dynamic through PHP and SQL database

simple web
  • Java Application Server

    • Errors as Java Stack Trace in the logs

con cluster. froscon

Today challenges

  • Java Application Server

    • Multiple JVM

    • Multiple Log Files

  • "Enterprise" SQL databases

  • Storage

  • SSO

    • SPNEGO / Kerberos

    • SAML

  • Load Balancer

  • Kubernetes

  • Elasticsearch

  • NoSQL, MongoDB

  • Redis

  • GraphQL

  • NodeJS

  • React

My daily work

connections netplan adv froscon

Responsible for the service and …​

connections netplan adv froscon hl

Check after installation and updates

  • Is the environment working as expected

    • Smoke testing

  • Does it scale for your planned user count

    • Load testing

pexels rafael guajardo 604684

Tools for Testing

K6 example

import encoding from 'k6/encoding';
import http from 'k6/http';
import { check } from 'k6';
const username = 'jjones3';
const password = 'password';

export const options = {
    insecureSkipTLSVerify: true
};
export default function () {
  const credentials = `${username}:${password}`;
  const url = `https://${credentials}@cnx7-rh8.stoeps.home/wikis/home`;
  let res = http.get(url);
  check(res, {
    'status is 200': (r) => r.status === 200,
    'LtpaToken2': (r) => r.headers.LtpaToken2 !== 0,
  });
}

K6 result - single user connect

k6 result1

k6 - GraphQL load testing

k6 without error
k6 with error

Selenium Example - Python

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService

options = webdriver.ChromeOptions()
options.set_capability('acceptSslCerts', True)
service = ChromeService(executable_path='/snap/chromium/1985/usr/lib/chromium-browser/chromedriver')
driver = webdriver.Chrome(service=service, options=options)

driver.maximize_window()
driver.get("https://cnx7-rh8.stoeps.home/wikis/login")
driver.find_element(by=By.ID, value="username").send_keys('jjones3')
driver.find_element(by=By.ID, value="password").send_keys("password")
driver.find_element(by=By.CLASS_NAME, value="lotusBtnSpecial").click()

Logs

Browser

First step on an error: disable the adblocker

Replace Fiddler with HAR

  • Webdeveloper Tools > Network, Enable persist logs, Reload the page

  • Right click > Save all as HAR

  • To analyze a recorded HAR, just drag&drop into your network tab

saveashar

Copy curl command line from WebDev Tools

copyascurl
curl 'https://cnx7-rh8.stoeps.home/social/api/mwgraphql' -X POST \
    -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0' \
    -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' \
    -H 'Content-Type: application/json' -H 'authorization: Bearer 82ee99648d5327[...]b507e1' \
    -H 'Origin: https://cnx7-rh8.stoeps.home' -H 'Connection: keep-alive' \
    -H 'Referer: https://cnx7-rh8.stoeps.home/homepage/' \
    -H 'Cookie: JSESSIONID=0000wjHhwi-[...]1fvsegm22; ROLE_metrics-report-run=false; ROLE_admin=false; lang=en; BAYEUX_BROWSER=ab06-15e90oty5xbqxl1mgaacexs6; ROLE_mail-user=true; blogsUser=Joe Jones2; LtpaToken2=mBsFxSGe5j[...]B5fqZtEiRiA==' -H 'Sec-Fetch-Dest: empty' \
    -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' \
    --data-raw '{"query":"query {userprefs {applications {orient_me {defaultHomeLink}}}}","variables":{}}'
HAR and curl command include authorization tokens

Javascript Debugger

20220818 161316

Javascript Debugger 2

20220818 161829
  1. Watch variables and expressions

  2. Breakpoint

  3. Step into the function

Javascript Debugger 3

20220818 162133
  • Variable substitution shown in source code

  • Step through the source with F11 and keep eye on Watch expressions

Debug SAML handshake

20220818 171558

SAML Tracer (Browser Addon)

20220818 170101
  • Decrypt the SAML data

  • Check mappings (uid, mail addresses)

Decrypted SAML POST

  • most of the time the wrong saml2:Attribute is configured

  • check response

<saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">
    <saml2:Attribute Name="email"
                NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified" >
        <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xsi:type="xs:string" >john.doe@example.com</saml2:AttributeValue>
    </saml2:Attribute>
</saml2:AttributeStatement>

Multi Account Container

  • Very convenient to open ISC and Connections with different users

    • ISC as user wasadmin, Connections as normal user

  • Test something with different users in one browser

    • e.g. create content and check notification

    • login with multiple accounts

  • Shares Cookies with containers of the same class

  • Private tab / window shares cookies of all private tabs

  • Container Proxy is an additional add-on

    • set proxies for containers of one kind

    • e.g. Burp Suite for one class, Tor for another

Multi Account Container — Container Proxy

containeraddon
containerproxy

Intercept proxies

  • Often used in Bug Bounty

  • Import HAR (ZAP)

  • See requests and responses

  • Bypass client side controls

  • Brute Force and Fuzz API or Logins

zap history
Figure 2. History
zap tree
Figure 3. Site tree

Intercept proxies example - OWASP ZAP

zap wide

Intercept proxies example - OWASP ZAP HUD

zap hud3

SSL

  • testssl.sh

    • Check SSL certificates

    • Like SSL Server Test

      • But faster

      • No need to publish your site to public

  • Keystore Explorer

    • Examine SSL

    • Import signer to TDI keystore

    • Convert certificates

    • Alternative Google all OpenSSL switches

testssl.sh — protocols and ciphers

testssl1
Figure 4. OpenLDAP
testssl4
Figure 5. nginx

testssl.sh — check application access

testssl3
Figure 6. OpenLDAP
testssl5
Figure 7. nginx

testssl.sh — check for vulnerabilities

testssl2

Keystore Explorer — Create keystore, import CA

  • Create Keystore (JKS)

  • Examine > Examine SSL

    • Add hostname & port

    • Import

20220818 172337
20220818 172406

Keystore Explorer — Create keystore, import CA

20220818 172649
20220818 172744

Keystore Explorer — Create keystore, import CA

20220818 172804
20220818 172821

Keystore Explorer — Create keystore, import CA

20220818 172937

Compare files

  • Compare files between production and testing

  • Side by side migration, compare directory trees

    • NEVER trust your documentation to find all changes

    • Or automate the deployment and configuration

  • Software

Meld

meld1

Ansible

  • Idempotent deployments

Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

  • Install tools on all servers with same paths and aliases

  • Commands on your fingertips in all environments

Linter

  • As standalone tools, or integrated into your favorite editor

    • VIM, Emacs

    • VS Code, notepad++

  • Examples

    • Ansible-lint

      • Ansible: Linting playbooks, roles and collections

    • yamllint

      • Kubernetes, Ansible

      • Autocomplete in editors, test in CI/CD pipeline

    • config-lint

      • Validates: Terraform, Kubernetes, LintRules, YAML, JSON

    • LanguageTool

      • Spellchecker, Grammar, Standalone, integrated in editor, browser add-on

Ansible Lint

ansible-lint playbooks
WARNING: PATH altered to include /usr/bin
WARNING  Listing 1 violation(s) that are fatal
syntax-check: couldn't resolve module/action 'xml'. This often indicates a misspelling,
missing collection, or incorrect module path.
roles/hcl/connections/clean_was_temp/tasks/main.yml:17:3 [WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
ERROR! couldn't resolve module/action 'xml'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-automation/roles/hcl/connections/clean_was_temp/tasks/main.yml': line 17, column 3,
but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name:                  Update versionStamp in LotusConnections-config.xml
  ^ here

Finished with 1 failure(s), 0 warning(s) on 71 files.

Ansible Lint

❯ ansible-lint -x yaml roles

[WARNING]: While constructing a mapping from /home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-
automation/roles/hcl/component-pack/tasks/setup_ingress.yml, line 45, column 3,
found a duplicate dict key (shell).
Using last defined value only.
[WARNING]: While constructing a mapping from <unicode string>, line 220, column 7,
found a duplicate dict key
(namespace). Using last defined value only.

fqcn-builtins: Use FQCN for builtin actions.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect

package-latest: Package installs should not use latest.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect

Finished with 2197 failure(s), 424 warning(s) on 628 files.

Yamllint

roles/third_party/ibm/db2-install/db2-restart/tasks/main.yml
  2:25      error    too many spaces after colon  (colons)
  3:25      error    too many spaces after colon  (colons)
  4:25      error    too many spaces after colon  (colons)
  9:25      warning  truthy value should be one of [false, true]  (truthy)
  9:81      error    line too long (158 > 80 characters)  (line-length)
  10:25     error    too many spaces after colon  (colons)
  11:25     error    too many spaces after colon  (colons)
  13:81     error    line too long (321 > 80 characters)  (line-length)
  15:4      error    wrong indentation: expected 4 but found 3  (indentation)

Decompiler (find issues)

  • jd-gui

    • Java Decompiler

  • Ghidra

    • A software reverse engineering (SRE) suite of tools developed by NSA

Icon java 64
ghidra

jd-gui

  • Example check Mobile APNS update file

jd gui1
  • Check ear-files, find trace settings

jd gui2

Ghidra

  • Decompiler

  • Check binaries for configuration strings

  • Example shows mod_ibm_upload.so

ghidra mod upload

Database (JDBC) Client — DBeaver Community

  • Supports DB2 and Elasticsearch

dbeaver4
dbeaver5
  • Converts binary ids (Files, Wikis) to UUID format for better readability

dbeaver1
dbeaver2
dbeaver3

Security

  • trivy

    • File System

    • Registry (Container)

  • Popeye

    • Scans live Kubernetes cluster

    • Detects misconfigurations

    • Helps you to ensure that best practices are in place

Trivy — container images

trivy image --input ~/vmware/software/cp_7.0.0.2/hybridcloud/images/admin-portal.tar -s CRITICAL

/home/stoeps/vmware/software/cp_7.0.0.2/hybridcloud/images/admin-portal.tar (alpine 3.12.0)
===========================================================================================
Total: 8 (CRITICAL: 8)

+----------+------------------+----------+-------------------+---------------+---------------------------------------+
|  LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |                 TITLE                 |
+----------+------------------+----------+-------------------+---------------+---------------------------------------+
|apk-tools | CVE-2021-36159   | CRITICAL | 2.10.5-r1         | 2.10.7-r0     | libfetch before 2021-07-26, as        |
|          |                  |          |                   |               | used in apk-tools, xbps, and          |
|          |                  |          |                   |               | other products, mishandles...         |
|          |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-36159 |
+----------+------------------+          +-------------------+---------------+---------------------------------------+
..
+----------+------------------+          +-------------------+---------------+---------------------------------------+
|ssl_client| CVE-2022-28391   |          | 1.31.1-r16        | 1.31.1-r22    | busybox: remote attackers may execute |
|          |                  |          |                   |               | arbitrary code if netstat is used     |
|          |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2022-28391 |
+----------+------------------+----------+-------------------+---------------+---------------------------------------+

Trivy — Filesystem

[root@cnx7-rh8-was ConnectionsCell]# trivy rootfs --ignore-unfixed -s CRITICAL Dogear.ear
[[C2022-05-18T10:18:07.039Z	INFO	Number of language-specific files: 1
2022-05-18T10:18:07.039Z	INFO	Detecting jar vulnerabilities...
2022-05-18T10:18:07.043Z	INFO	Table result includes only package filenames. Use '--format json' option to get the full path to the package file.

Java (jar)
Total: 4 (CRITICAL: 4)
┌───────────────────────────────────────────┬──────────────────┬──────────┬────────────┬────────────┬──────────────────────────────────────────────────────────────┐
│                           Library         │  Vulnerability   │ Severity │ Installed V│ Fixed Versi│                            Title                             │
├───────────────────────────────────────────┼──────────────────┼──────────┼────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ com.googlecode.owasp-java-html-sanitizer: │ CVE-2021-42575   │ CRITICAL │ 20171016.1 │ 20211018.1 │ owasp-java-html-sanitizer: improper policies enforcement may │
│ owasp-java-html-sa-nitizer (owasp-java-   │                  │          │            │            │ lead to remote code execution                                │
│ html-sanitizer-20171016.1.jar)            │                  │          │            │            │ https://avd.aquasec.com/nvd/cve-2021-42575                   │
├───────────────────────────────────────────┼──────────────────┤          ├────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ commons-collections:commons-collections   │ CVE-2015-7501    │          │ 3.2.1      │ 3.2.2      │ apache-commons-collections: InvokerTransformer code          │
│ (commons-collections-3.2.1.jar)           │                  │          │            │            │ execution during deserialisation                             │
│                                           │                  │          │            │            │ https://avd.aquasec.com/nvd/cve-2015-7501                    │
├───────────────────────────────────────────┼──────────────────┼──────────┼────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ commons-fileupload:commons-fileupload     │ CVE-2016-1000031 │ CRITICAL │ 1.2.1      │ 1.3.3      │ Apache Commons FileUpload: DiskFileItem file manipulation    │
│ (commons-fileupload-1.2.1.jar)            │                  │          │            │            │ https://avd.aquasec.com/nvd/cve-2016-1000031                 │
└───────────────────────────────────────────┴──────────────────┴──────────┴────────────┴────────────┴──────────────────────────────────────────────────────────────┘

popeye

popeye

Kubernetes

  • Some tools to work faster on the console

  • Get logs from running pods

  • Speed up the most used tasks with kubectl

    • K9s — (Win, Mac, Lnx)

  • Get some insights of the blackbox Kubernetes

kubectx, kubens

  • Change context or namespace for kubectl

  • kubectl ns shows all namespaces

    • select the namespace to set a new default

  • Speed up kubectl

    • no need to type -n connections over and over again

  • kubectx or kubectl ctx

    • set Kubernetes master and user to connect with kubectl

    • useful if you administrate multiple Kubernetes clusters from one host

kubens
kubectx interactive
kubectx

kubetail, stern

  • kubetail

    • Display logs from multiple containers/pods

    • Regular Expression or label to select

  • stern

    • Shows running container, not all

    • -c selects container

kubetail
stern

k9s

  • Replaces watch kubectl get pods for me

  • Check logs of pods and containers from the terminal ui

  • No complicated cli commands necessary

Istio

  • Componentpack is somehow a black box

  • No documentation on dependencies

    • Which pods should I check when for example Orient Me isn’t working

    • Which pods can you restart without affecting Customizer?

  • Istio

    • Service Mesh

      • Traffic Management

      • Observability

      • Security

Istio — Installation

Fast, but unsecure
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.13.4 TARGET_ARCH=x86_64 sh -
cd istio-1.13.4
export PATH=$PWD/bin:$PATH
istioctl install --set profile=demo
istioctl manifest apply --set components.cni.enabled=true

kubectl get pods -n istio-system

NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-fgrhx                    1/1     Running   0          168m
istio-ingressgateway-76dcc86449-5z9rd   1/1     Running   0          168m
istiod-7664dfcb67-5wsgz                 1/1     Running   0          168m

Istio — Sidecar

Enable sidecars for namespace connections
kubectl label namespace connections istio-injection=enabled
Disable sidecar for Elasticsearch
kubectl edit statefulsets.apps/es-data-7
kubectl edit statefulsets.apps/es-master-7
kubectl edit deployment es-client-7
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
  • Restart all statefulsets and deployments

Install Kiali on top of Istio

  • Kiali is an observability console for Istio

    • With service mesh configuration and validation capabilities

  • Helps you understand the structure and health of your service mesh

kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.13/samples/addons/kiali.yaml

kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.13/samples/addons/prometheus.yaml

Documentation

  • I write most documention in

  • pandoc converts from any of these formats to (e.g.)

    • HTML

    • PDF

    • MS Word

    • …​

  • txt based formats can be version controlled in git

Container

  • Work with local container

    • Use tools

    • Test

    • Change images

  • Podman

  • Docker

  • Dive

    • Analyze container layer and changes

Some more useful tools for daily work

❯ echo $JSON | gron
json = [];
json[0] = {};
json[0].id = "003";
json[0].name = "John Brooks";
json[1] = {};
json[1].id = "053";
json[1].name = "Randy Park";

❯ echo $JSON | gron | grep name
json[0].name = "John Brooks";
json[1].name = "Randy Park";
  • JQ

    • Great tool, syntax not self explaining

❯ echo $JSON | jq
[
  {
    "name": "John Brooks",
    "id": "003"
  },
  {
    "name": "Randy Park",
    "id": "053"
  }
]

❯ echo $JSON | jq '.[].name'
"John Brooks"
"Randy Park"