Greg Heartsfield
c8e8b71b91
fix: use accurate timer for slow queries, and use 250ms as cutoff
2023-01-31 18:09:43 -06:00
Greg Heartsfield
6d57adef73
improvement: log filter in a reusable format for slow queries
2023-01-31 18:09:43 -06:00
Greg Heartsfield
111eb4a10c
perf: prevent sqlite readers from capturing worker thread pool and impacting writer latency
2023-01-31 18:09:23 -06:00
Greg Heartsfield
214f152c5d
improvement: provide reason for abort in prometheus metric
2023-01-30 18:40:47 -06:00
Greg Heartsfield
3fcaf97a15
improvement: move sqlite connection acquisition into blocking thread
2023-01-30 18:02:40 -06:00
Greg Heartsfield
cec501b37f
improvement: start timing for each new filter execution
2023-01-30 18:02:40 -06:00
Greg Heartsfield
2557c7f69c
improvement: run filters as separate queries, to reduce complexity on SQLite query planner
2023-01-30 18:02:40 -06:00
Greg Heartsfield
3979a94726
improvement: do not force query to use index when limit is specified
2023-01-30 18:02:40 -06:00
Greg Heartsfield
71bdbfb425
improvement: query and exit early for events that get immediately replaced
2023-01-30 18:02:40 -06:00
Greg Heartsfield
b6798f96b6
improvement: add prometheus metrics, renaming others
2023-01-30 18:02:28 -06:00
w3irdrobot
c1152ce430
improvement(NIP-19): identify and parse NIP-19 addresses
2023-01-29 18:55:30 -06:00
thesimplekid
6f1a4e7d76
fix: postgres create index before column exists
2023-01-29 18:32:42 -06:00
Greg Heartsfield
1804bee912
feat(NIP-33): parameterized replaceable events for postgres
2023-01-29 18:30:08 -06:00
Greg Heartsfield
34db91940c
improvement: prometheus metrics for aborted queries
2023-01-28 16:05:58 -06:00
Greg Heartsfield
0859e535ed
improvement: remove origin label from connections metric
...
The origin header is controlled by clients, and there is no expiration
of any values that appear. We would need to whitelist a set of known
origins in order to track this without giving someone the ability to
exhaust memory.
2023-01-28 15:46:26 -06:00
Greg Heartsfield
bdd4e43df4
improvement: show errors when writing new sqlite db
...
Build the sqlite writer pool first, which will provide a better error
message in the event the database is not writeable or readable.
2023-01-28 14:02:20 -06:00
Greg Heartsfield
57e1b53c13
feat: postgres migration schema v2
...
This primarily deals with correctly handling tag values.
2023-01-27 20:13:47 -06:00
Greg Heartsfield
53f83aa923
improvement: delete, do not hide, replaceable events
2023-01-24 08:04:42 -06:00
Greg Heartsfield
c8f7420334
feat(NIP-33): parameterized replaceable events
2023-01-24 08:04:42 -06:00
Greg Heartsfield
e2869e8fad
fix(NIP-16): do not replace events unless they are newer
2023-01-24 08:04:42 -06:00
Greg Heartsfield
5c07b2eca5
refactor: event is_ephemeral method
2023-01-24 08:04:42 -06:00
Greg Heartsfield
25752abe6b
fix: run postgres migration on startup
2023-01-24 08:04:37 -06:00
Kieran
16f6e974c8
feat: add support for PostgreSQL as a backend repository
2023-01-22 16:26:54 -06:00
Kieran
4121c872bc
feat: prometheus metrics
...
Prometheus metrics exposed at /metrics
2023-01-22 11:08:12 -06:00
Greg Heartsfield
6489e685ab
refactor: reformat and remove tabs
2023-01-22 10:06:44 -06:00
Greg Heartsfield
6800c2e39d
improvement: add NostrRepo trait, with sqlite implementation
...
This is inspired by the work of
v0l (https://github.com/v0l/nostr-rs-relay/ ).
A new trait abstracts the storage layer with an async API. Rusqlite
is still used with worker threads, but this allows for Postgresql or
other backends to be used.
There may be bugs, this has not been rigorously tested.
2023-01-22 09:49:49 -06:00
Greg Heartsfield
585fdd3884
fix: use data_dir from config.toml if present
...
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/64
2023-01-16 17:21:12 -06:00
Greg Heartsfield
13cc24b5cd
improvement: log blacklisted events
2023-01-15 15:42:27 -06:00
Greg Heartsfield
f543957b34
improvement: clear out hidden events during schema upgrade
2023-01-15 15:27:41 -06:00
Greg Heartsfield
7021f102e8
improvement: delete replaceable events
2023-01-15 15:13:10 -06:00
Greg Heartsfield
fddbf321bc
perf: add indexes and force their use (authors)
2023-01-15 10:52:49 -06:00
Greg Heartsfield
3e7f2e21df
perf: force authors index to be used if possible
2023-01-15 10:23:46 -06:00
Greg Heartsfield
9d9c6c78d1
improvement: refuse to insert events that would automatically be hidden
2023-01-15 10:01:01 -06:00
Greg Heartsfield
703b2efe6e
refactor: replaceable check in event
2023-01-15 09:18:53 -06:00
Greg Heartsfield
0db6487ce3
fix: allow tokio tracing to be enabled
...
fixes https://github.com/scsibug/nostr-rs-relay/issues/48
2023-01-14 09:47:23 -06:00
Rasmus Schlunsen
73f4f60cc7
improvement: use clap for command line args
2023-01-14 09:22:11 -06:00
Greg Heartsfield
d06d227ebe
improvement: lower REQ logging and note possible truncation
2023-01-11 16:56:40 -06:00
Greg Heartsfield
3519488c4e
improvement: lower logging for failed REQ parses
2023-01-10 07:41:49 -06:00
Greg Heartsfield
fbd3315110
improvement: log REQ messages at debug level
2023-01-09 22:12:20 -06:00
Greg Heartsfield
3d3d1bde53
refactor: clippy suggestions
2023-01-09 22:12:04 -06:00
Greg Heartsfield
ed336111bb
improvement: alert before long-running migration
2023-01-09 22:11:25 -06:00
Greg Heartsfield
62e8da689d
fix: do not force kind_created_at_index when there are tags
2023-01-06 12:57:48 -06:00
Greg Heartsfield
807d1aa384
improvement: log index names used
2023-01-06 12:50:52 -06:00
Greg Heartsfield
66a55b55b9
perf: new index, manually selected when appropriate
2023-01-06 12:17:30 -06:00
Greg Heartsfield
76c77c3e56
feat: bulk loading script for importing events
2023-01-06 12:16:19 -06:00
Greg Heartsfield
50daab8a6f
refactor: make a standalone re-tagging function
2023-01-06 06:57:56 -06:00
Greg Heartsfield
e9d2a2cbd0
perf(NIP-26): temporarily disable NIP-26 delegated events
2023-01-04 16:51:22 -06:00
Greg Heartsfield
39a945b493
perf: separate author/delegated_by queries, minor improvement
2023-01-04 16:51:17 -06:00
Greg Heartsfield
9a84dc19e9
perf: author/kind index added (schema v13)
2023-01-04 16:51:02 -06:00
Greg Heartsfield
20c4bb42eb
fix: correct log message
2023-01-03 21:24:46 -06:00
JesterHodl
0e519f6b77
feat: add --help and --version flags
...
fixes: https://github.com/scsibug/nostr-rs-relay/issues/42
2023-01-03 17:39:21 -06:00
Greg Heartsfield
3dd0f2c9c6
fix: do not run auto_vacuum on read-only connections
2023-01-03 17:32:55 -06:00
Greg Heartsfield
b7c8737166
improvement: enable auto_vacuum on database creation
2023-01-03 06:22:43 -06:00
Greg Heartsfield
c0b112c094
improvement: enable auto_vacuum on connections
2023-01-03 06:22:04 -06:00
Greg Heartsfield
cb283ac316
fix: ensure that replaceable events are handled correctly regardless of order receieved
2023-01-02 17:18:11 -06:00
Greg Heartsfield
2c6ac69bfd
docs: remove incorrect comment
2023-01-02 15:41:17 -06:00
Greg Heartsfield
d929ae2752
improvement: define websocket send queue (unlimited->1024)
2023-01-02 15:39:28 -06:00
Greg Heartsfield
14fe9f9ee1
improvement: remove pauses for backups, likely not needed w/ WAL compaction
2023-01-02 15:38:30 -06:00
0xtr
7774db8c47
feat: add event kind blacklist
...
Adds a list to the config where you can specify which event kinds to blacklist.
The blacklist check will run right after verifying that the pubkey is allowed
to post events to the relay.
2022-12-27 17:10:34 -06:00
Greg Heartsfield
19ec89593d
improvement: drop queries that are running during a checkpoint
2022-12-27 15:24:10 -06:00
Greg Heartsfield
27902bc5f4
improvement: move reader mutex closer to DB connection acquisition
2022-12-27 10:28:56 -06:00
Greg Heartsfield
d2adddaee4
improvement: extend allowed wal_checkpoint timeout to 10 sec
2022-12-27 10:13:14 -06:00
Greg Heartsfield
b23b3ce8ec
improvement: block new readers when WAL is large
2022-12-27 09:48:07 -06:00
Greg Heartsfield
5f9fe1ce59
improvement: do not send realtime only filters to the DB (limit:0)
2022-12-26 12:20:36 -06:00
Greg Heartsfield
65fd0ed08b
feat: increase wal_checkpoint time when WAL is large
2022-12-26 10:03:51 -06:00
Greg Heartsfield
0b51675b38
improvement: change suggestion and default for max sqlite DB readers
2022-12-25 11:17:08 -06:00
Greg Heartsfield
2e22334631
refactor: formatting
2022-12-25 11:06:30 -06:00
Greg Heartsfield
cb2ac4bf0f
improvement: give threads unique names
2022-12-25 10:47:32 -06:00
Greg Heartsfield
38dc7789dc
improvement: cleaner slow query logs
2022-12-25 10:47:32 -06:00
Greg Heartsfield
ce0e00ffb3
feat: log reader DB pool stats every minute
2022-12-25 10:47:32 -06:00
Greg Heartsfield
3e4ae4aeec
feat: cache prepared statements and trace expanded SQL queries
2022-12-25 10:47:32 -06:00
Greg Heartsfield
c6a8807485
improvement: send error on empty-string prefix author/id searches
2022-12-25 10:47:32 -06:00
Greg Heartsfield
8137b6211c
refactor: clippy suggestions
2022-12-24 10:29:47 -06:00
Greg Heartsfield
e5074f2e46
feat(NIP-28): replaceable kind 41 channel metadata events
2022-12-24 10:14:43 -06:00
Blake Jakopovic
4fd7643907
feat: change pub(crate) to pub for use as a library
2022-12-23 07:14:58 -06:00
benthecarman
e08647867c
refactor: remove code duplication for simple_event
2022-12-23 06:39:50 -06:00
Greg Heartsfield
4f1a912f36
feat: log origin header from websocket requests
...
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/33
2022-12-22 16:55:53 -06:00
Greg Heartsfield
25480e837f
fix: do not block writers for more than 1 second during checkpoints
2022-12-22 16:10:49 -06:00
Greg Heartsfield
b80b54cd9d
improvement: reduce logging, especially for database pool size
2022-12-22 15:47:33 -06:00
Greg Heartsfield
8ea732cbe5
feat: perform regular database maintenance (60sec), without blocking main writer thread
2022-12-22 15:16:21 -06:00
Greg Heartsfield
0f68c4e5c2
refactor: formatting
2022-12-22 15:15:45 -06:00
Greg Heartsfield
f411aa6fc2
fix: do not re-verify NIP-05 entries where metadata was deleted
2022-12-22 13:01:48 -06:00
Greg Heartsfield
d31bbda087
improvement: reduce lifetime of database connections
2022-12-22 13:01:12 -06:00
Greg Heartsfield
5917bc53b2
improvement: run maintenance every 60 seconds instead of by event count
2022-12-22 11:40:17 -06:00
Greg Heartsfield
91177c61a1
improvement: log reason for new event creation from nip05
2022-12-22 10:48:30 -06:00
Greg Heartsfield
53c2a8051c
improvement: reduce logging
2022-12-22 10:29:27 -06:00
Greg Heartsfield
168cf513ac
feat: perform full checkpoints and truncate WAL every 2k events
2022-12-22 10:11:05 -06:00
Greg Heartsfield
ea204761c9
fix: do not show slow queries more than once per sub
2022-12-20 15:41:50 -06:00
Greg Heartsfield
c270ae1434
improvement: reduce event count for db writer pauses
2022-12-20 15:25:24 -06:00
Greg Heartsfield
64bd983cb6
perf: every 5000 persisted events, pause for 500ms for backups
...
I have observed backups running for a very long time under heavy load,
this introduces some artificial delay to give the online backup enough
time to make progress.
2022-12-20 15:05:04 -06:00
Greg Heartsfield
1c153bc784
perf: shed DB query load when queue gets large
2022-12-20 13:23:21 -06:00
Greg Heartsfield
dc11d9a619
improvement: explicitly rollback transaction on duplicate event
2022-12-20 13:23:04 -06:00
Greg Heartsfield
cd1557787b
improvement: log write pool
2022-12-20 13:21:57 -06:00
Greg Heartsfield
86bb7aeb9a
improvement: function to check pool capacity
2022-12-20 10:07:01 -06:00
Greg Heartsfield
2cfd384339
perf: drop db handles that are not quickly read
2022-12-19 00:18:39 -06:00
Greg Heartsfield
8c013107f9
perf: increase upper bound for sqlite mmap
2022-12-18 23:19:43 -06:00
Greg Heartsfield
64a4466d30
perf: backing down on max_blocking_threads
2022-12-18 23:14:41 -06:00
Greg Heartsfield
1596c23eb4
perf: increase blocking threads now that contention is reduced
2022-12-18 22:46:32 -06:00
Greg Heartsfield
129badd4e1
perf: reduce per thread mmap allocation for DB
2022-12-18 22:45:32 -06:00
Greg Heartsfield
6f7c080180
improvement: reduce number of writer blocking threads from 4->2
2022-12-18 22:32:31 -06:00
Greg Heartsfield
af92561ef6
perf: remove shared cache mode (experiment)
2022-12-18 22:15:50 -06:00
Greg Heartsfield
d833a3e40d
perf: reduce logging
2022-12-18 22:11:46 -06:00
Greg Heartsfield
cf144d503d
perf: reduce logging for slow queries
2022-12-18 20:47:11 -06:00
Greg Heartsfield
88ac31b549
perf: increase channel size for DB communication
2022-12-18 13:44:28 -06:00
Greg Heartsfield
677b7d39e9
improvement: log slow requests that return zero results
2022-12-18 13:42:31 -06:00
Greg Heartsfield
b24d2f9aaa
perf: set default blocking threads to lower value
2022-12-18 12:20:57 -06:00
Greg Heartsfield
d10348f7e1
feat: configurable blocking threads
2022-12-18 09:14:04 -06:00
Greg Heartsfield
8598e443d8
wip: add configuration for future feature (client concurrent db limits)
2022-12-17 23:19:48 -06:00
Greg Heartsfield
43222d44e5
feat: perform optimization after seeing many events
2022-12-17 23:18:54 -06:00
Greg Heartsfield
7c1516c4fb
perf: add index for tags
2022-12-17 23:17:53 -06:00
Greg Heartsfield
0c72053a49
perf: increase mmap size to 1GB
2022-12-17 23:17:16 -06:00
Greg Heartsfield
3f32ff67ab
improvement: minor logging
2022-12-17 23:11:14 -06:00
Greg Heartsfield
0b9778d6ca
refactor: simplify tracking of subscriptions
2022-12-17 20:46:58 -06:00
Greg Heartsfield
cc06167e06
perf: add composite index for tag table
2022-12-17 12:01:20 -06:00
Greg Heartsfield
b6e33f044f
improvement: limit db connection max lifetime
2022-12-17 10:47:35 -06:00
Greg Heartsfield
0d8d39ad22
feat: add rate limiting setting for subscription creation
2022-12-17 09:27:29 -06:00
Greg Heartsfield
3c880b2f49
perf: pull distinct to outermost SQL
2022-12-17 07:49:28 -06:00
Greg Heartsfield
7a4c9266ec
improvement: make hexsearch structs sortable
2022-12-17 07:49:05 -06:00
Greg Heartsfield
7ca9c864f2
improvement: DB pool logging shows used connections directly
2022-12-16 17:01:49 -06:00
Greg Heartsfield
838aafd079
improvement: consistent log messages for client/sub ids
2022-12-16 15:22:27 -06:00
Greg Heartsfield
e554b10ac2
improvement: tweak sub/sql logging for slow queries
2022-12-16 14:55:45 -06:00
Greg Heartsfield
b0bfaa48fc
improvement: ignore duplicate REQ messages
2022-12-16 14:37:02 -06:00
Greg Heartsfield
2e9b1b6ba7
docs: comment reason for force_no_match
2022-12-16 14:35:21 -06:00
Greg Heartsfield
f9695bd0a9
fix: db schema version updates correctly for v9
2022-12-16 10:01:49 -06:00
Greg Heartsfield
7c4bf5cc8f
fix: run db migration for v9
2022-12-16 08:21:00 -06:00
Greg Heartsfield
e2de162931
feat: only show SQL in logs for slow queries unless tracing
2022-12-16 08:17:39 -06:00
Greg Heartsfield
4f606615eb
perf: indexing improvement
2022-12-16 08:16:49 -06:00
Greg Heartsfield
c48e45686d
perf: schema updates for better event indexing
2022-12-15 08:48:35 -06:00
Greg Heartsfield
bbe359364a
refactor: clippy warnings
2022-12-15 08:43:36 -06:00
Greg Heartsfield
9e9c494367
perf: significant query speedup when using kinds.
...
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/51
2022-12-14 21:04:49 -06:00
Greg Heartsfield
5fa24bc9f1
fix: send EOSE when ids list is empty in subscriptions
...
Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/50
2022-11-19 10:35:00 -06:00
Greg Heartsfield
4de7490d97
fix: send EOSE when authors list is empty in subscriptions
...
Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/49
2022-11-19 10:00:38 -06:00
Greg Heartsfield
110500bb46
feat(NIP-20): advertise support for NIP-20 in relay info/readme
2022-11-12 09:22:43 -06:00
Greg Heartsfield
83f6b11de7
refactor: clippy fix
2022-11-12 09:22:24 -06:00
William Casarin
6d1244434b
feat(NIP-20): improve invalid event error messages
...
Instead of returning a NOTICE for invalid events, return a `OK false`
command result with a reason as to why the event is invalid.
2022-11-12 09:13:22 -06:00
William Casarin
5a91419d34
feat(NIP-20): send command results to clients
...
When submitting events to relays, clients currently have no way to know
if an event was successfully committed to the database. This NIP
introduces the concept of command results which are like NOTICE's except
provide more information about if an event was accepted or rejected.
A command result is a JSON object with the following structure that is
returned when an event is successfully saved to the database or
rejected:
["OK", <event_id>, <true|false>, <message>]
nip20: https://github.com/nostr-protocol/nips/pull/62
2022-11-12 09:12:35 -06:00
William Casarin
7adc5c9af7
perf: dont create intermediate vecs when matching subs
...
Avoid creating intermediate vectors when matching subscriptions. We can
just iterate over the hashmap directly.
2022-11-09 07:30:43 -06:00
Greg Heartsfield
9dd4571bee
refactor: reduce level of some common DB logs
2022-11-06 13:49:32 -06:00
Greg Heartsfield
9db5a26b9c
refactor: more consistent logging messages
2022-11-05 16:11:20 -05:00
Greg Heartsfield
ac345b5744
refactor: do not quote server-generated client id in logs
2022-11-05 15:59:39 -05:00
Greg Heartsfield
5a8860bb09
feat: log user-agent if present
2022-11-05 10:29:25 -05:00
Greg Heartsfield
11e43eccf9
refactor: add unit to ping_interval config
2022-11-05 07:42:08 -05:00
William Casarin
50577b2dfa
feat: add network.ping_interval setting
...
Add a ping interval setting that allows you to customize the websocket
ping interval. The default of 5 minutes may be too high for some proxy
servers that disconnect connections that are held open for too long.
2022-11-05 07:40:28 -05:00
William Casarin
a6cb6f8486
refactor: rename get_header_remote_ip -> get_header_string
...
This function has nothing to do with remote ips!
2022-11-05 07:37:18 -05:00
Greg Heartsfield
ae5bf98d87
feat: retrieve client IP from header in config.toml
...
If the config.toml has defined a HTTP header to look for a remote IP,
that will be logged. Otherwise, the socket address IP will be used.
closes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/47
2022-11-04 18:05:01 -05:00
William Casarin
1cf9d719f0
feat: look for proxied ip headers
...
This enables support for using the proxied IP from cloudflare. The damus
relay is behind cloudflare, so to get accurate remote ip logging we need
to look at the headers instead of the socket address.
Signed-off-by: William Casarin <jb55@jb55.com>
2022-11-04 17:09:28 -05:00
William Casarin
311f4b5283
refactor: switch new connections to debug log
...
These are pretty spammy on busy relays. I've been using the info log to
monitor spam attacks, and these are the least useful info log.
Leave the "stopping connection" log because it at least provides useful
sent/received information.
Signed-off-by: William Casarin <jb55@jb55.com>
2022-11-04 07:59:53 -05:00
Greg Heartsfield
14b5a51e3a
fix: log ephemeral events after send
2022-11-04 07:55:38 -05:00
Greg Heartsfield
8ecce3f566
feat: show client IP in logs
2022-11-02 18:33:44 -05:00
Greg Heartsfield
72f8a1aa5c
feat(NIP-26): allow searches for delegated public keys
...
Implements core NIP-26 delegated event functionality. Events can
include a `delegation` tag that provides a signature and restrictions
on which events can be delegated.
Notable points on the implementation so far:
* Schema has been upgraded to include an index and new column.
* Basic rune parsing/evaluation to implement the example event in the
NIP, but no more.
* No special logic for deletion.
* No migration logic for determining delegated authors for
already-stored events.
2022-10-16 15:25:06 -05:00
