Experience in "Enterprise" Application Server
Migrations, Deployments
Performance Analysis, Infrastructure
Focusing in
Monitoring, Security
Troubleshooting
More and more
DevOps, Cloud, Automation
Support Open Source developers!
Tons of options, contribute, test, document, or recommend tools
If we want to see a lively open source scene, we need to keep actively using open source software, and not be afraid of trying out new ones.
If we find something good, we shouldn’t hesitate to recommend it to others, so they can also benefit from it.
Complete collection: https://stoeps.de/tiu
Short description
License
Most of the tools shown in this presentation
Are open source
Available for Linux, Mac OS and Windows
Just a web server
dynamic through PHP and SQL database
Java Application Server
Errors as Java Stack Trace in the logs
Java Application Server
Multiple JVM
Multiple Log Files
"Enterprise" SQL databases
Storage
SSO
SPNEGO / Kerberos
SAML
Load Balancer
Kubernetes
Elasticsearch
NoSQL, MongoDB
Redis
GraphQL
NodeJS
React
Is the environment working as expected
Smoke testing
Does it scale for your planned user count
Load testing
K6
Windows, Linux, macOS, Docker
JavaScript, import HAR
Load testing, chaos and reliability testing, performance testing
Selenium
Automates browsers (needs Chrome webdriver, Firefox webdriver)
Cross browser & smoke testing
Apache JMeter
import encoding from 'k6/encoding';
import http from 'k6/http';
import { check } from 'k6';
const username = 'jjones3';
const password = 'password';
export const options = {
insecureSkipTLSVerify: true
};
export default function () {
const credentials = `${username}:${password}`;
const url = `https://${credentials}@cnx7-rh8.stoeps.home/wikis/home`;
let res = http.get(url);
check(res, {
'status is 200': (r) => r.status === 200,
'LtpaToken2': (r) => r.headers.LtpaToken2 !== 0,
});
}
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
options = webdriver.ChromeOptions()
options.set_capability('acceptSslCerts', True)
service = ChromeService(executable_path='/snap/chromium/1985/usr/lib/chromium-browser/chromedriver')
driver = webdriver.Chrome(service=service, options=options)
driver.maximize_window()
driver.get("https://cnx7-rh8.stoeps.home/wikis/login")
driver.find_element(by=By.ID, value="username").send_keys('jjones3')
driver.find_element(by=By.ID, value="password").send_keys("password")
driver.find_element(by=By.CLASS_NAME, value="lotusBtnSpecial").click()
Log file analytics, traces
Find the right trace strings
ELK (Elasticsearch, Logstash, Kibana)
Open source real-time web log analyzer
Interactive viewer that runs in a terminal
DevTools (Firebug, Chrome Web Developer)
HAR Analytics (HTTP Archive)
Copy as curl
Javascript debugger (needs some developer knowledge)
Addons
Intercept Proxy
First step on an error: disable the adblocker |
Webdeveloper Tools > Network, Enable persist logs
, Reload the page
Right click > Save all as HAR
To analyze a recorded HAR, just drag&drop into your network tab
curl
command line from WebDev Toolscurl 'https://cnx7-rh8.stoeps.home/social/api/mwgraphql' -X POST \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0' \
-H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' \
-H 'Content-Type: application/json' -H 'authorization: Bearer 82ee99648d5327[...]b507e1' \
-H 'Origin: https://cnx7-rh8.stoeps.home' -H 'Connection: keep-alive' \
-H 'Referer: https://cnx7-rh8.stoeps.home/homepage/' \
-H 'Cookie: JSESSIONID=0000wjHhwi-[...]1fvsegm22; ROLE_metrics-report-run=false; ROLE_admin=false; lang=en; BAYEUX_BROWSER=ab06-15e90oty5xbqxl1mgaacexs6; ROLE_mail-user=true; blogsUser=Joe Jones2; LtpaToken2=mBsFxSGe5j[...]B5fqZtEiRiA==' -H 'Sec-Fetch-Dest: empty' \
-H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' \
--data-raw '{"query":"query {userprefs {applications {orient_me {defaultHomeLink}}}}","variables":{}}'
HAR and curl command include authorization tokens |
Watch variables and expressions
Breakpoint
Step into the function
Variable substitution shown in source code
Step through the source with F11 and keep eye on Watch expressions
Decrypt the SAML data
Check mappings (uid, mail addresses)
most of the time the wrong saml2:Attribute
is configured
check response
<saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">
<saml2:Attribute Name="email"
NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified" >
<saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:type="xs:string" >john.doe@example.com</saml2:AttributeValue>
</saml2:Attribute>
</saml2:AttributeStatement>
Very convenient to open ISC and Connections with different users
ISC as user wasadmin, Connections as normal user
Test something with different users in one browser
e.g. create content and check notification
login with multiple accounts
Shares Cookies with containers of the same class
Private tab / window shares cookies of all private tabs
Container Proxy is an additional add-on
set proxies for containers of one kind
e.g. Burp Suite for one class, Tor for another
Often used in Bug Bounty
Import HAR (ZAP)
See requests and responses
Bypass client side controls
Brute Force and Fuzz API or Logins
Check SSL certificates
Like SSL Server Test
But faster
No need to publish your site to public
Examine SSL
Import signer to TDI keystore
Convert certificates
Alternative Google all OpenSSL switches
testssl.sh
— protocols and cipherstestssl.sh
— check application accesstestssl.sh
— check for vulnerabilitiesCreate Keystore (JKS)
Examine > Examine SSL
Add hostname & port
Import
Idempotent deployments
Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.
Install tools on all servers with same paths and aliases
Commands on your fingertips in all environments
As standalone tools, or integrated into your favorite editor
VIM, Emacs
VS Code, notepad++
Examples
Ansible: Linting playbooks, roles and collections
Kubernetes, Ansible
Autocomplete in editors, test in CI/CD pipeline
Validates: Terraform, Kubernetes, LintRules, YAML, JSON
Spellchecker, Grammar, Standalone, integrated in editor, browser add-on
ansible-lint playbooks
WARNING: PATH altered to include /usr/bin
WARNING Listing 1 violation(s) that are fatal
syntax-check: couldn't resolve module/action 'xml'. This often indicates a misspelling,
missing collection, or incorrect module path.
roles/hcl/connections/clean_was_temp/tasks/main.yml:17:3 [WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
ERROR! couldn't resolve module/action 'xml'. This often indicates a misspelling, missing collection, or incorrect module path.
The error appears to be in '/home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-automation/roles/hcl/connections/clean_was_temp/tasks/main.yml': line 17, column 3,
but may be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: Update versionStamp in LotusConnections-config.xml
^ here
Finished with 1 failure(s), 0 warning(s) on 71 files.
❯ ansible-lint -x yaml roles
[WARNING]: While constructing a mapping from /home/stoeps/ghq/github.com/HCL-TECH-SOFTWARE/connections-
automation/roles/hcl/component-pack/tasks/setup_ingress.yml, line 45, column 3,
found a duplicate dict key (shell).
Using last defined value only.
[WARNING]: While constructing a mapping from <unicode string>, line 220, column 7,
found a duplicate dict key
(namespace). Using last defined value only.
fqcn-builtins: Use FQCN for builtin actions.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect
package-latest: Package installs should not use latest.
roles/third_party/tiny-editors-install/tasks/setup_os.yml:26 Task/Handler: Install Pexpect
Finished with 2197 failure(s), 424 warning(s) on 628 files.
roles/third_party/ibm/db2-install/db2-restart/tasks/main.yml
2:25 error too many spaces after colon (colons)
3:25 error too many spaces after colon (colons)
4:25 error too many spaces after colon (colons)
9:25 warning truthy value should be one of [false, true] (truthy)
9:81 error line too long (158 > 80 characters) (line-length)
10:25 error too many spaces after colon (colons)
11:25 error too many spaces after colon (colons)
13:81 error line too long (321 > 80 characters) (line-length)
15:4 error wrong indentation: expected 4 but found 3 (indentation)
jd-gui
Example check Mobile APNS update file
Check ear-files, find trace settings
Decompiler
Check binaries for configuration strings
Example shows mod_ibm_upload.so
Analyzed configuration parameters
Documentation missing
Supports DB2 and Elasticsearch
Converts binary ids (Files, Wikis) to UUID format for better readability
trivy image --input ~/vmware/software/cp_7.0.0.2/hybridcloud/images/admin-portal.tar -s CRITICAL
/home/stoeps/vmware/software/cp_7.0.0.2/hybridcloud/images/admin-portal.tar (alpine 3.12.0)
===========================================================================================
Total: 8 (CRITICAL: 8)
+----------+------------------+----------+-------------------+---------------+---------------------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION | TITLE |
+----------+------------------+----------+-------------------+---------------+---------------------------------------+
|apk-tools | CVE-2021-36159 | CRITICAL | 2.10.5-r1 | 2.10.7-r0 | libfetch before 2021-07-26, as |
| | | | | | used in apk-tools, xbps, and |
| | | | | | other products, mishandles... |
| | | | | | -->avd.aquasec.com/nvd/cve-2021-36159 |
+----------+------------------+ +-------------------+---------------+---------------------------------------+
..
+----------+------------------+ +-------------------+---------------+---------------------------------------+
|ssl_client| CVE-2022-28391 | | 1.31.1-r16 | 1.31.1-r22 | busybox: remote attackers may execute |
| | | | | | arbitrary code if netstat is used |
| | | | | | -->avd.aquasec.com/nvd/cve-2022-28391 |
+----------+------------------+----------+-------------------+---------------+---------------------------------------+
[root@cnx7-rh8-was ConnectionsCell]# trivy rootfs --ignore-unfixed -s CRITICAL Dogear.ear
[[C2022-05-18T10:18:07.039Z INFO Number of language-specific files: 1
2022-05-18T10:18:07.039Z INFO Detecting jar vulnerabilities...
2022-05-18T10:18:07.043Z INFO Table result includes only package filenames. Use '--format json' option to get the full path to the package file.
Java (jar)
Total: 4 (CRITICAL: 4)
┌───────────────────────────────────────────┬──────────────────┬──────────┬────────────┬────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Installed V│ Fixed Versi│ Title │
├───────────────────────────────────────────┼──────────────────┼──────────┼────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ com.googlecode.owasp-java-html-sanitizer: │ CVE-2021-42575 │ CRITICAL │ 20171016.1 │ 20211018.1 │ owasp-java-html-sanitizer: improper policies enforcement may │
│ owasp-java-html-sa-nitizer (owasp-java- │ │ │ │ │ lead to remote code execution │
│ html-sanitizer-20171016.1.jar) │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2021-42575 │
├───────────────────────────────────────────┼──────────────────┤ ├────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ commons-collections:commons-collections │ CVE-2015-7501 │ │ 3.2.1 │ 3.2.2 │ apache-commons-collections: InvokerTransformer code │
│ (commons-collections-3.2.1.jar) │ │ │ │ │ execution during deserialisation │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2015-7501 │
├───────────────────────────────────────────┼──────────────────┼──────────┼────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│ commons-fileupload:commons-fileupload │ CVE-2016-1000031 │ CRITICAL │ 1.2.1 │ 1.3.3 │ Apache Commons FileUpload: DiskFileItem file manipulation │
│ (commons-fileupload-1.2.1.jar) │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2016-1000031 │
└───────────────────────────────────────────┴──────────────────┴──────────┴────────────┴────────────┴──────────────────────────────────────────────────────────────┘
popeye
kubectx
, kubens
Change context or namespace for kubectl
kubectl ns
shows all namespaces
select the namespace to set a new default
Speed up kubectl
no need to type -n connections
over and over again
kubectx
or kubectl ctx
set Kubernetes master and user to connect with kubectl
useful if you administrate multiple Kubernetes clusters from one host
kubetail
, stern
kubetail
Display logs from multiple containers/pods
Regular Expression or label to select
stern
Shows running container, not all
-c
selects container
k9s
Replaces watch kubectl get pods
for me
Check logs of pods and containers from the terminal ui
No complicated cli commands necessary
Componentpack is somehow a black box
No documentation on dependencies
Which pods should I check when for example Orient Me isn’t working
Which pods can you restart without affecting Customizer?
Istio
Service Mesh
Traffic Management
Observability
Security
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.13.4 TARGET_ARCH=x86_64 sh -
cd istio-1.13.4
export PATH=$PWD/bin:$PATH
istioctl install --set profile=demo
istioctl manifest apply --set components.cni.enabled=true
kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-cni-node-fgrhx 1/1 Running 0 168m
istio-ingressgateway-76dcc86449-5z9rd 1/1 Running 0 168m
istiod-7664dfcb67-5wsgz 1/1 Running 0 168m
kubectl label namespace connections istio-injection=enabled
kubectl edit statefulsets.apps/es-data-7
kubectl edit statefulsets.apps/es-master-7
kubectl edit deployment es-client-7
spec:
template:
metadata:
annotations:
sidecar.istio.io/inject: "false"
Restart all statefulsets and deployments
Kiali is an observability console for Istio
With service mesh configuration and validation capabilities
Helps you understand the structure and health of your service mesh
kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.13/samples/addons/kiali.yaml
kubectl apply -f \
https://raw.githubusercontent.com/istio/istio/release-1.13/samples/addons/prometheus.yaml
I write most documention in
pandoc
converts from any of these formats to (e.g.)
HTML
MS Word
…
txt based formats can be version controlled in git
❯ echo $JSON | gron
json = [];
json[0] = {};
json[0].id = "003";
json[0].name = "John Brooks";
json[1] = {};
json[1].id = "053";
json[1].name = "Randy Park";
❯ echo $JSON | gron | grep name
json[0].name = "John Brooks";
json[1].name = "Randy Park";
Great tool, syntax not self explaining
❯ echo $JSON | jq
[
{
"name": "John Brooks",
"id": "003"
},
{
"name": "Randy Park",
"id": "053"
}
]
❯ echo $JSON | jq '.[].name'
"John Brooks"
"Randy Park"
Session Slides
Tools Listing
Regular update