Observability - Revision history

R389li: /* Metrics */ make Alerts section and add Grafana alerts option

2024-05-27T02:14:56Z

Metrics: make Alerts section and add Grafana alerts option

← Older revision		Revision as of 21:14, 26 May 2024
Line 27:		Line 27:
	)		)
	</pre>		</pre>

	<br>
			== Alerts ==
	We also use [https://prometheus.io/docs/alerting/latest/alertmanager/ AlertManager] to send email alerts from Prometheus metrics. We should figure out how to also send messages to IRC or similar.		We also use [https://prometheus.io/docs/alerting/latest/alertmanager/ AlertManager] to send email alerts from Prometheus metrics. We should figure out how to also send messages to IRC or similar.

Line 33:		Line 34:

	We make some pretty charts on Grafana (https://prometheus.csclub.uwaterloo.ca) from PromQL queries. Grafana also has an 'Explorer' page where you can test out some queries before making chart panels from them.		We make some pretty charts on Grafana (https://prometheus.csclub.uwaterloo.ca) from PromQL queries. Grafana also has an 'Explorer' page where you can test out some queries before making chart panels from them.

			You can also create fine-grained alerts on Grafana (https://prometheus.csclub.uwaterloo.ca/alerting/list).

	== Logging ==		== Logging ==

2023-12-16T02:06:50Z

Schema

← Older revision		Revision as of 21:06, 15 December 2023
Line 122:		Line 122:
	GRANT ALL ON vector.* TO vector;		GRANT ALL ON vector.* TO vector;
	CREATE USER grafana IDENTIFIED BY 'REPLACE_ME' SETTINGS PROFILE 'readonly';		CREATE USER grafana IDENTIFIED BY 'REPLACE_ME' SETTINGS PROFILE 'readonly';
	GRANT SHOW DATABASES, SHOW TABLES, SELECT ON . TO grafana;		GRANT SHOW DATABASES, SHOW TABLES, SELECT, DICTGET ON . TO grafana;
	</pre>		</pre>

2023-12-16T02:05:16Z

ClickHouse

@@ Line 100: / Line 100: @@
   <mysql_port></mysql_port>
   <postgresql_port></postgresql_port>
 </clickhouse>
 </pre>

2023-12-16T02:04:04Z

ClickHouse

@@ Line 61: / Line 61: @@
 <pre>
 <clickhouse>
-  <readonly>
+  <profiles>
   <users>
     <default>

2023-12-16T01:23:40Z

ClickHouse

← Older revision		Revision as of 20:23, 15 December 2023
Line 113:		Line 113:
	Then run <code>clickhouse-client</code> and create the dictionary:		Then run <code>clickhouse-client</code> and create the dictionary:
	<pre>		<pre>
	CREATE DICTIONARY country_codes_dictionary		CREATE DICTIONARY vector.country_codes_dictionary
	(		(
	Name String,		Name String,

2023-12-16T00:06:09Z

Don't remove country_codes.csv

← Older revision		Revision as of 19:06, 15 December 2023
Line 126:		Line 126:
	<pre>		<pre>
	SELECT * FROM country_codes_dictionary;		SELECT * FROM country_codes_dictionary;
	</pre>
	Now exit the SQL shell and delete the CSV file since we don't need it anymore:
	<pre>
	rm /var/lib/clickhouse/user_files/country_codes.csv
	</pre>		</pre>

2023-11-03T04:57:51Z

Schema

@@ Line 233: / Line 233: @@
 </pre>
-Finally, we'll also keep stats just for the university:
+We also want to keep stats just for the university:
 <pre>
 CREATE TABLE vector.mirror_requests_agg_uw
@@ Line 266: / Line 266: @@
    OR isIPAddressInRange(IPv6NumToString(ip_address), 'fd74:6b6a:8eca::/47')
 GROUP BY distro, date, region_name, city;
 </pre>

2023-10-14T18:27:51Z

ClickHouse

← Older revision		Revision as of 13:27, 14 October 2023
Line 81:		Line 81:
	</clickhouse>		</clickhouse>
	</pre>		</pre>
	Then paste the following into /etc/clickhouse-server/config/~~csclub~~-~~config~~.xml:		Then paste the following into /etc/clickhouse-server/config/zzz-csclub.xml (we need the zzz prefix because the configuration files are merged in alphabetical order, and we want ours to be applied last):
	<pre>		<pre>
	<clickhouse>		<clickhouse>
			<listen_host>127.0.0.1</listen_host>
			<listen_host>::1</listen_host>
	<logger>		<logger>
	<level>information</level>		<level>information</level>

2023-10-14T18:05:14Z

Vector

← Older revision		Revision as of 13:05, 14 October 2023
Line 386:		Line 386:
	status = parsed_log.status		status = parsed_log.status
	request = string!(parsed_log.request \|\| "")		request = string!(parsed_log.request \|\| "")
	if is_null(err) && status == 200 ~~&& !contains(request, "/..")~~ {		if is_null(err) && status == 200 {
	parsed_path, err = parse_regex(request, r'^GET /+(?P<distro>[^/? ]+)')		parsed_path, err = parse_regex(request, r'^GET /+(?P<distro>[^/? ]+)')
	distro = parsed_path.distro		distro = parsed_path.distro
Line 394:		Line 394:
	"pool", "dists", "csclub.asc", "csclub.gpg"		"pool", "dists", "csclub.asc", "csclub.gpg"
	]		]
			if (
	if is_null(err) && !includes(ignore, distro) {		is_null(err) && !includes(ignore, distro) && !contains(request, "..") &&
			!starts_with(request, "#") && !starts_with(request, "%") && !starts_with(request, ".")
			) {
	. = {		. = {
	"distro": distro,		"distro": distro,

2023-10-14T17:36:14Z

Schema

← Older revision		Revision as of 12:36, 14 October 2023
Line 143:		Line 143:
	ENGINE = MergeTree()		ENGINE = MergeTree()
	PRIMARY KEY (host, timestamp)		PRIMARY KEY (host, timestamp)
	TTL timestamp + INTERVAL 3 MONTH DELETE;		TTL timestamp + INTERVAL 1 MONTH DELETE;
	</pre>		</pre>