參考的網站

前言

因為計畫需求,我們本有做一套基於風險的存取控制模型,但是該模型要滿足一個要求,需要能夠監控裝置,當裝置發現異常時,要能夠告警並通知一個貼標籤的服務,那存取控制模型就可以基於標籤和存取策略,適當的拒絕不滿足存取策略的請求

本文章的前提

因此本篇文章的目的如下:

  • [ ] 了解wazuh這套工具是如何做告警?
  • [ ] 如果wazuh告警,發送請求該怎麼做?

關於可以作為告警的工具

  • 我想要的功能是:如果滿足特定規則 (e.g. level5以上的alert) 可以做到發送 mail 和 call api
  • 目前我找到的工具:
    • 工具1 Watcher : elasticsearch 本身有 plugin 可以支援,稱為 elasticSearch Watcher 可以做到上述需求,但是並非開源。
    • 工具2 elastAlert: 所以目前想使用 elasticAlert 這個開源的的專案來進行。

具體來說,做告警的方式,就是不斷地去問 elasticSearch 然後如果發現滿足的要求或是條件,就執行某些動作。因為 wazuh 是把 log 透過 filebeat 傳到 elasticSearch 進行儲存,因此我們可以著重在當收到 elasticSearch 的特定訊息時,執行某些動作。

elasticAlert 介紹

目前 elasticAlert 已經不再維護,以 elasticAlert2 為主,主要更可靠、高度模塊化且易於設置和配置。

參考:https://elastalert2.readthedocs.io/en/latest/elastalert.html

ElasticAlert 可以提供常見的監控規則,像是:

  • 特定時間內發生事件的頻率(frequent type)
  • 當事件發出率提高或下降時(spike type)
  • 事件在時間內觸發少於幾次(filatline type)
  • 滿足白名單或黑名單時(blacklist and whitelist type)
  • 滿足任何特定filter時(any type)
  • 在特定時間內,某個欄位值改變時(change type)

當滿足上述情境時,可以做到像是:Email, Http Post, Jira, Line Notify等等,並且在通知的告警中添加alert link、或是相關資料而我們想要的是Email和Http Post功能,因此本篇教學也會著重在實作這兩個部分。

安裝教學:使用 docker 啟動

參考官方: https://elastalert2.readthedocs.io/en/latest/running_elastalert.html#as-a-docker-container

官方我認為沒寫到很完整,因為我還有設置 elasticSearch 需要透過 TLS/SSH 來進行連線,因此比起官網需要多一些設置。

請注意:在這邊你會需要使用到 elasticSearch 的 ca.crt,並且請確保該檔案的權限設定為任何人都可以讀取如下:
chmod 777 ca.crt

  1. 首先你會先需要如下的檔案結構
1
2
3
4
5
6
7
8
9
# 我是在 /opt/elastalert 底下進行
mkdir /opt/elastalert

.
├── certs
│   └── ca.crt # 這是 elasticSearch 的 ca.crt 請移動過來 並且進行 chmod 777 否則會出現
├── elastalert.yaml
└── rules
└── a.yml
  1. 開始建立 /opt/elastalert/ealstalert.yaml 檔案,我是參考官網建議的檔案:https://github.com/jertel/elastalert2/blob/master/examples/config.yaml.example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
# This is the folder that contains the rule yaml files
# This can also be a list of directories
# Any .yaml file will be loaded as a rule
rules_folder: /opt/elastalert/rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: 192.168.2.71

# The Elasticsearch port
es_port: 9200

# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1

# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Optional prefix for statsd metrics
#statsd_instance_tag: elastalert

# Optional statsd host
#statsd_host: dogstatsd

# Connect with TLS to Elasticsearch
use_ssl: True

# Verify TLS certificates
verify_certs: True

# Show TLS or certificate related warnings
ssl_show_warn: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See https://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
es_username: elastic
es_password: 4RQwIyVNCBznV4zXGDhX

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
ca_certs: /opt/elastalert/certs/ca.crt
#client_cert: /etc/elasticsearch/certs/elasticsearch.crt
#client_key: /etc/elasticsearch/certs/elasticsearch.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 1

建立 /opt/rules/a.yaml 檔案

在這之前你可能會好奇webhook要填寫什麼,你可以透過以下教學建立 webhook
可以參考這篇:https://api.slack.com/messaging/webhooks

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
name: "Shannon-Alert-Test"
type: "frequency"
index: "wazuh-*"
is_enabled: true
buffer_time: # 查詢 elasticsearch 的時間區間
minutes: 5

# 5分鐘內相同的報警不會重複發送
realert:
minutes: 1
terms_size: 50 # 每個查詢返回的最大數量 default 50

# 頻率觸發設定
# 當 48 小時內,滿足 authorization
num_events: 1 # 將觸發警報的事件數量(含)
timeframe: # 在 timeframe 時間內內必須發生 num_events 次數
hours: 64
timestamp_field: "@timestamp"
timestamp_type: "iso"
use_strftime_index: false #

# 警告訊息
alert_subject: "Alert! {0} ! AgentID: {1} in time ({2})"
alert_subject_args:
- "rule.groups"
- "agent.id"
- "@timestamp"
alert_text: "主要原因: {0}"
alert_text_args:
- "full_log"

# 用於查找事件的 Elasticsearch 過濾器列表
filter:
- query:
query_string:
query: "@timestamp:*"
- query:
wildcard:
rule.groups: "*authentication_fail*"

# 透過什麼方式進行通知
alert:
- "slack"
slack_webhook_url: 'https://hooks.slack.com/services/xxxxx' # 請使用你自己的 webhook url
slack_msg_color: "warning"
slack_parse_override: "none"

透過 docker 指令啟動

啟動,跟官方的有一點不一樣,因為我們需要把 ca.crt mount 到容器得的 /opt/elastalert/certs/ca.crt,因為 ealstalert.yaml 是這樣設定的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 先前往 /opt/elastalert/ 
cd /opt/elastalert

# pwd 會直接帶目前路徑
# 你會發現我們把3個部分mount進去,分別是 elastalert.yaml, rules, ca.crt
docker run --net=es_default -d --name elastalert --restart=always \
-v $(pwd)/elastalert.yaml:/opt/elastalert/config.yaml \
-v $(pwd)/rules:/opt/elastalert/rules \
-v $(pwd)/certs:/opt/elastalert/certs \
jertel/elastalert2 --verbose

# 確認 logs
docker logs -f elastalert

# 輸出結果應該要類似如下
INFO:elastalert:Background configuration change check run at 2023-07-02 17:47 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2023-07-02 17:47 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.99996 seconds
INFO:elastalert:Queried rule a from 2023-07-02 17:33 UTC to 2023-07-02 17:48 UTC: 0 / 0 hits
WARNING:elasticsearch:DELETE https://192.168.2.71:9200/_search/scroll [status:404 request:0.001s]
INFO:elastalert:Ran a from 2023-07-02 17:33 UTC to 2023-07-02 17:48 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 900

透過 docker-compose 啟動

你也可以建立 /opt/elastalert/docker-compose.yaml 檔案,內容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
version: '3.4'
services:
elastalert:
container_name: elastalert2
image: jertel/elastalert2
volumes:
- /opt/elastalert/config/config.yaml:/opt/elastalert/config.yaml
- /opt/elastalert/rules:/opt/elastalert/rules
- /opt/elastalert/certs:/opt/elastalert/certs
command: --verbose
restart: always
networks:
- es_default
ports:
- "3030:3030"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3030"]
interval: 30s
timeout: 15s
retries: 3
start_period: 40s

networks:
es_default:
external: true

然後透過以下指令啟動

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 先前往 /opt/elastalert/
cd /opt/elastalert
# 啟動 docker-compose.yaml 背景執行
docker-compose up -d
# 查看 logs
docker-compose logs -f elastalert
# 輸出結果應該要類似如下
INFO:elastalert:Background configuration change check run at 2023-07-02 17:47 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2023-07-02 17:47 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.99996 seconds
INFO:elastalert:Queried rule a from 2023-07-02 17:33 UTC to 2023-07-02 17:48 UTC: 0 / 0 hits
WARNING:elasticsearch:DELETE https://192.168.2.71:9200/_search/scroll [status:404 request:0.001s]
INFO:elastalert:Ran a from 2023-07-02 17:33 UTC to 2023-07-02 17:48 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 900

總結

回到文章的開頭想探討的,到目前為止總結如下:

  • [x] 了解wazuh這套工具是如何做告警?
    • Ans: Wazuh 本身提供 mail 發訊息功能,但是如果要做更多還是使用 elastic 整合相關告警工具更好
  • [x] 如果wazuh告警,發送請求該怎麼做?
    • Ans: 目前看到的做法是 (1) elastic 內建的 watcher 但是因為非開源,加上還需要license,因此選擇 (2) elastAlert 開源工具。

接下來東西串好了,elastAlert 可以正常使用,我會優先透過 slack 來測試 Alert 的相關功能

坑1: SSLError Permission Denied

Ans: 本來一直出現以下問題,後來發現是ca.crt的權限讓elastAlert無法讀取。
只需要進行 chmod 777 ca.crt 後 再mount到container即可解決

1
2
3
File "/usr/local/lib/python3.11/site-packages/elasticsearch/connection/http_requests.py", line 174, in perform_request
raise SSLError("N/A", str(e), e)
elasticsearch.exceptions.SSLError: ConnectionError(HTTPSConnectionPool(host='192.168.2.71', port=9200): Max retries exceeded with url: / (Caused by SSLError(PermissionError(13, 'Permission denied')))) caused by: SSLError(HTTPSConnectionPool(host='192.168.2.71', port=9200): Max retries exceeded with url: / (Caused by SSLError(PermissionError(13, 'Permission denied'))))