Greg Heartsfield
5917bc53b2
improvement: run maintenance every 60 seconds instead of by event count
2022-12-22 11:40:17 -06:00
Greg Heartsfield
91177c61a1
improvement: log reason for new event creation from nip05
2022-12-22 10:48:30 -06:00
Greg Heartsfield
53c2a8051c
improvement: reduce logging
2022-12-22 10:29:27 -06:00
Greg Heartsfield
168cf513ac
feat: perform full checkpoints and truncate WAL every 2k events
2022-12-22 10:11:05 -06:00
Greg Heartsfield
ea204761c9
fix: do not show slow queries more than once per sub
2022-12-20 15:41:50 -06:00
Greg Heartsfield
c270ae1434
improvement: reduce event count for db writer pauses
2022-12-20 15:25:24 -06:00
Greg Heartsfield
64bd983cb6
perf: every 5000 persisted events, pause for 500ms for backups
...
I have observed backups running for a very long time under heavy load,
this introduces some artificial delay to give the online backup enough
time to make progress.
2022-12-20 15:05:04 -06:00
Greg Heartsfield
1c153bc784
perf: shed DB query load when queue gets large
2022-12-20 13:23:21 -06:00
Greg Heartsfield
dc11d9a619
improvement: explicitly rollback transaction on duplicate event
2022-12-20 13:23:04 -06:00
Greg Heartsfield
cd1557787b
improvement: log write pool
2022-12-20 13:21:57 -06:00
Greg Heartsfield
86bb7aeb9a
improvement: function to check pool capacity
2022-12-20 10:07:01 -06:00
Greg Heartsfield
2cfd384339
perf: drop db handles that are not quickly read
2022-12-19 00:18:39 -06:00
Greg Heartsfield
8c013107f9
perf: increase upper bound for sqlite mmap
2022-12-18 23:19:43 -06:00
Greg Heartsfield
64a4466d30
perf: backing down on max_blocking_threads
2022-12-18 23:14:41 -06:00
Greg Heartsfield
1596c23eb4
perf: increase blocking threads now that contention is reduced
2022-12-18 22:46:32 -06:00
Greg Heartsfield
129badd4e1
perf: reduce per thread mmap allocation for DB
2022-12-18 22:45:32 -06:00
Greg Heartsfield
6f7c080180
improvement: reduce number of writer blocking threads from 4->2
2022-12-18 22:32:31 -06:00
Greg Heartsfield
af92561ef6
perf: remove shared cache mode (experiment)
2022-12-18 22:15:50 -06:00
Greg Heartsfield
d833a3e40d
perf: reduce logging
2022-12-18 22:11:46 -06:00
Greg Heartsfield
cf144d503d
perf: reduce logging for slow queries
2022-12-18 20:47:11 -06:00
Greg Heartsfield
88ac31b549
perf: increase channel size for DB communication
2022-12-18 13:44:28 -06:00
Greg Heartsfield
677b7d39e9
improvement: log slow requests that return zero results
2022-12-18 13:42:31 -06:00
Greg Heartsfield
b24d2f9aaa
perf: set default blocking threads to lower value
2022-12-18 12:20:57 -06:00
Greg Heartsfield
d10348f7e1
feat: configurable blocking threads
2022-12-18 09:14:04 -06:00
Greg Heartsfield
8598e443d8
wip: add configuration for future feature (client concurrent db limits)
2022-12-17 23:19:48 -06:00
Greg Heartsfield
43222d44e5
feat: perform optimization after seeing many events
2022-12-17 23:18:54 -06:00
Greg Heartsfield
7c1516c4fb
perf: add index for tags
2022-12-17 23:17:53 -06:00
Greg Heartsfield
0c72053a49
perf: increase mmap size to 1GB
2022-12-17 23:17:16 -06:00
Greg Heartsfield
3f32ff67ab
improvement: minor logging
2022-12-17 23:11:14 -06:00
Greg Heartsfield
0b9778d6ca
refactor: simplify tracking of subscriptions
2022-12-17 20:46:58 -06:00
Greg Heartsfield
cc06167e06
perf: add composite index for tag table
2022-12-17 12:01:20 -06:00
Greg Heartsfield
b6e33f044f
improvement: limit db connection max lifetime
2022-12-17 10:47:35 -06:00
Greg Heartsfield
0d8d39ad22
feat: add rate limiting setting for subscription creation
2022-12-17 09:27:29 -06:00
Greg Heartsfield
3c880b2f49
perf: pull distinct to outermost SQL
2022-12-17 07:49:28 -06:00
Greg Heartsfield
7a4c9266ec
improvement: make hexsearch structs sortable
2022-12-17 07:49:05 -06:00
Greg Heartsfield
7ca9c864f2
improvement: DB pool logging shows used connections directly
2022-12-16 17:01:49 -06:00
Greg Heartsfield
838aafd079
improvement: consistent log messages for client/sub ids
2022-12-16 15:22:27 -06:00
Greg Heartsfield
e554b10ac2
improvement: tweak sub/sql logging for slow queries
2022-12-16 14:55:45 -06:00
Greg Heartsfield
b0bfaa48fc
improvement: ignore duplicate REQ messages
2022-12-16 14:37:02 -06:00
Greg Heartsfield
2e9b1b6ba7
docs: comment reason for force_no_match
2022-12-16 14:35:21 -06:00
Greg Heartsfield
f9695bd0a9
fix: db schema version updates correctly for v9
2022-12-16 10:01:49 -06:00
Greg Heartsfield
7c4bf5cc8f
fix: run db migration for v9
2022-12-16 08:21:00 -06:00
Greg Heartsfield
e2de162931
feat: only show SQL in logs for slow queries unless tracing
2022-12-16 08:17:39 -06:00
Greg Heartsfield
4f606615eb
perf: indexing improvement
2022-12-16 08:16:49 -06:00
Greg Heartsfield
c48e45686d
perf: schema updates for better event indexing
2022-12-15 08:48:35 -06:00
Greg Heartsfield
bbe359364a
refactor: clippy warnings
2022-12-15 08:43:36 -06:00
Greg Heartsfield
9e9c494367
perf: significant query speedup when using kinds.
...
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/51
2022-12-14 21:04:49 -06:00
Greg Heartsfield
5fa24bc9f1
fix: send EOSE when ids list is empty in subscriptions
...
Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/50
2022-11-19 10:35:00 -06:00
Greg Heartsfield
4de7490d97
fix: send EOSE when authors list is empty in subscriptions
...
Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/49
2022-11-19 10:00:38 -06:00
Greg Heartsfield
110500bb46
feat(NIP-20): advertise support for NIP-20 in relay info/readme
2022-11-12 09:22:43 -06:00
Greg Heartsfield
83f6b11de7
refactor: clippy fix
2022-11-12 09:22:24 -06:00
William Casarin
6d1244434b
feat(NIP-20): improve invalid event error messages
...
Instead of returning a NOTICE for invalid events, return a `OK false`
command result with a reason as to why the event is invalid.
2022-11-12 09:13:22 -06:00
William Casarin
5a91419d34
feat(NIP-20): send command results to clients
...
When submitting events to relays, clients currently have no way to know
if an event was successfully committed to the database. This NIP
introduces the concept of command results which are like NOTICE's except
provide more information about if an event was accepted or rejected.
A command result is a JSON object with the following structure that is
returned when an event is successfully saved to the database or
rejected:
["OK", <event_id>, <true|false>, <message>]
nip20: https://github.com/nostr-protocol/nips/pull/62
2022-11-12 09:12:35 -06:00
William Casarin
7adc5c9af7
perf: dont create intermediate vecs when matching subs
...
Avoid creating intermediate vectors when matching subscriptions. We can
just iterate over the hashmap directly.
2022-11-09 07:30:43 -06:00
Greg Heartsfield
9dd4571bee
refactor: reduce level of some common DB logs
2022-11-06 13:49:32 -06:00
Greg Heartsfield
9db5a26b9c
refactor: more consistent logging messages
2022-11-05 16:11:20 -05:00
Greg Heartsfield
ac345b5744
refactor: do not quote server-generated client id in logs
2022-11-05 15:59:39 -05:00
Greg Heartsfield
5a8860bb09
feat: log user-agent if present
2022-11-05 10:29:25 -05:00
Greg Heartsfield
11e43eccf9
refactor: add unit to ping_interval config
2022-11-05 07:42:08 -05:00
William Casarin
50577b2dfa
feat: add network.ping_interval setting
...
Add a ping interval setting that allows you to customize the websocket
ping interval. The default of 5 minutes may be too high for some proxy
servers that disconnect connections that are held open for too long.
2022-11-05 07:40:28 -05:00
William Casarin
a6cb6f8486
refactor: rename get_header_remote_ip -> get_header_string
...
This function has nothing to do with remote ips!
2022-11-05 07:37:18 -05:00
Greg Heartsfield
ae5bf98d87
feat: retrieve client IP from header in config.toml
...
If the config.toml has defined a HTTP header to look for a remote IP,
that will be logged. Otherwise, the socket address IP will be used.
closes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/47
2022-11-04 18:05:01 -05:00
William Casarin
1cf9d719f0
feat: look for proxied ip headers
...
This enables support for using the proxied IP from cloudflare. The damus
relay is behind cloudflare, so to get accurate remote ip logging we need
to look at the headers instead of the socket address.
Signed-off-by: William Casarin <jb55@jb55.com>
2022-11-04 17:09:28 -05:00
William Casarin
311f4b5283
refactor: switch new connections to debug log
...
These are pretty spammy on busy relays. I've been using the info log to
monitor spam attacks, and these are the least useful info log.
Leave the "stopping connection" log because it at least provides useful
sent/received information.
Signed-off-by: William Casarin <jb55@jb55.com>
2022-11-04 07:59:53 -05:00
Greg Heartsfield
14b5a51e3a
fix: log ephemeral events after send
2022-11-04 07:55:38 -05:00
Greg Heartsfield
8ecce3f566
feat: show client IP in logs
2022-11-02 18:33:44 -05:00
Greg Heartsfield
72f8a1aa5c
feat(NIP-26): allow searches for delegated public keys
...
Implements core NIP-26 delegated event functionality. Events can
include a `delegation` tag that provides a signature and restrictions
on which events can be delegated.
Notable points on the implementation so far:
* Schema has been upgraded to include an index and new column.
* Basic rune parsing/evaluation to implement the example event in the
NIP, but no more.
* No special logic for deletion.
* No migration logic for determining delegated authors for
already-stored events.
2022-10-16 15:25:06 -05:00
Greg Heartsfield
3e8adf978f
refactor: move db migrations into isolated functions
2022-10-09 08:54:03 -05:00
Greg Heartsfield
2af5f9fbe8
fix: correct schema upgrade logic (and refactor)
...
Schema upgrades were buggy from 4->5 (the v5 would be skipped). This
change also refactors the logic slightly so that future additions can
be clearer (no need to have if and else-if combinations).
2022-10-09 08:24:01 -05:00
Greg Heartsfield
2739e49362
fix: correct future schema version detection
2022-10-08 13:15:48 -05:00
Greg Heartsfield
f9693f7ac3
fix(NIP-9): hide events received after their deletions
...
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/40
2022-10-08 12:12:41 -05:00
Greg Heartsfield
8a63d88b0b
fix: prevent deletion of deletion events
2022-10-08 08:02:16 -05:00
Greg Heartsfield
92da9d71f8
feat: handle and log TERM signals
2022-09-28 07:20:31 -05:00
Greg Heartsfield
6633f8b472
feat: replace logging with tracing
2022-09-28 07:19:59 -05:00
Greg Heartsfield
93dfed0a87
refactor: misc clippy suggestions
2022-09-24 19:28:02 -05:00
Greg Heartsfield
bef7ca7e27
refactor: misc clippy suggestions
2022-09-24 09:19:16 -05:00
Greg Heartsfield
a98708ba47
refactor: misc clippy suggestions
2022-09-24 09:01:09 -05:00
Greg Heartsfield
ccf9b8d47b
refactor: remove unnecessary return types
2022-09-24 08:39:41 -05:00
Greg Heartsfield
8fa58de49a
refactor: clippy suggestions
2022-09-24 08:30:22 -05:00
Greg Heartsfield
4fa8616c73
feat: enable use of tokio-console with diagnostics.tracing setting
...
View real-time tokio diagnostics by setting the configuration option
"diagnostics.tracing" to true.
2022-09-11 12:44:45 -05:00
Greg Heartsfield
74802522c2
improvement: do not create NIP-05 thread if feature is disabled
2022-09-11 11:01:36 -05:00
Greg Heartsfield
9ce5057af8
improvement: better log formatting
2022-09-11 10:22:01 -05:00
Greg Heartsfield
78da92ccca
feat: advertise support for NIP-09 and NIP-12 in relay info
...
NIP-01 prefix search, and NIP-12 generic tags are no longer marked as
experimental.
NIP-11 relay info advertises NIP-09 event deletion and NIP-12 generic
tag search support.
2022-09-10 20:45:09 -05:00
Greg Heartsfield
72f1c19b21
feat(NIP-22): advertise support for event created_at limits
...
The `reject_future_limits` option can now be disabled, and is by
default.
NIP-11 advertises support for created_at limits.
The message for future-dated events has been modified, to be closer to
the recommended example in the NIP.
2022-09-10 20:40:10 -05:00
Greg Heartsfield
08b011ad07
feat: ensure that WAL is truncated after checkpoint
2022-09-10 19:18:57 -05:00
Greg Heartsfield
2b03f11e5e
refactor: remove global/singleton settings object
2022-09-06 06:12:07 -05:00
Greg Heartsfield
e48bae10e6
feat: support in-memory SQLite database
2022-09-06 06:06:01 -05:00
Greg Heartsfield
8774416b92
refactor: move nostr server into library
2022-09-06 05:56:04 -05:00
Greg Heartsfield
1b9f364e15
chore: rustfmt
2022-09-02 12:38:31 -05:00
Greg Heartsfield
a754477a02
improvement: misc refactorings (clippy)
2022-09-02 12:26:00 -05:00
Greg Heartsfield
a843eaa939
improvement: db.rs from clippy
2022-09-02 10:30:51 -05:00
Greg Heartsfield
03a130b0b8
improvement: simplify config builder (clippy)
2022-09-02 10:18:16 -05:00
slaninas
77892b2064
fix: syntax error
2022-08-22 05:12:52 -07:00
Greg Heartsfield
4fe6191aa3
chore: formatting
2022-08-21 09:51:34 -07:00
Greg Heartsfield
79a982e3ef
improvement: send NOTICE for too-large messages
2022-08-21 09:28:31 -07:00
Greg Heartsfield
01d81db617
improvement: log client id for subscription removal
2022-08-21 09:11:38 -07:00
Greg Heartsfield
e6fef37d4e
chore: rustfmt
2022-08-21 09:10:19 -07:00
Greg Heartsfield
035cf34673
fix(NIP-12): correctly search for mixed-case hex-like tags
...
Only lowercase and even-length tag values are stored as binary BLOBs.
Previously there was an error which search results from being returned
if the tag value was mixed-case and could be interpreted as hex.
A new database migration has been created to repair the `tag` table
for existing relays.
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/37
2022-08-17 16:34:11 -07:00
Greg Heartsfield
be8170342e
fix(NIP-12): multi-tag searches returns correct results
...
Logic of generated SQL was incorrect, causing multiple tag searches
(as defined in NIP-12) to produce no results.
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/36
2022-08-11 22:16:10 -07:00
Greg Heartsfield
0a3b15f41f
fix(NIP-11): Add CORS header and content type for main page
2022-08-11 19:33:17 -07:00
Greg Heartsfield
5058d98ad6
fix(NIP-12): only allow single-char tag filters
2022-08-07 10:15:36 -05:00
Greg Heartsfield
1c14adc766
fix(NIP-01): allow limits on a per-filter basis
...
The original implementation of subscription limit applied to the
entire query, instead of the specific filter. Now, each filter gets
its own query limit. When a limit is applied, the most recent N
events will be returned, otherwise the default is to return the
earliest events (in order), for all matching events.
2022-07-04 17:25:32 -05:00
Semisol
168cfc3b26
feat(NIP-16): Implement NIP16
...
NIP16 introduces a replaceable and ephemeral event range:
[10000..20000) for replaceable and [20000..30000) for
ephemeral.
2022-05-30 21:43:06 -05:00
Semisol
a36ad378f6
feat(NIP-15): Implement NIP15
...
NIP15 sends an EOSE notice to clients after all stored events are sent
to allow loading indicators and other use cases.
2022-05-30 21:43:00 -05:00
Greg Heartsfield
9ed3391b46
fix(NIP-09): correct WHERE clause for event deletion
2022-05-10 16:50:52 -05:00
William Casarin
4ad483090e
feat(NIP-01): Implement limit
...
This was quickly sneaked in by fiatjaf per my request[0], it makes many
queries more efficient and allows for paging when combined with until.
It is a bit weird to have multiple limits on each filter... for now we
just choose any or the last limit seen.
[0]: a4aea5337f
2022-05-10 16:47:56 -05:00
Greg Heartsfield
597749890e
improvement: remove unnecessary event logging
2022-02-27 19:30:48 -06:00
Greg Heartsfield
1d499cf12b
feat: handle NIP-09 for deletion events
2022-02-27 11:35:23 -06:00
Greg Heartsfield
ed3a6b9692
refactor: simplify NOTICE messages
2022-02-26 17:34:58 -06:00
Greg Heartsfield
414e83f696
refactor: import cleanup for config
2022-02-26 11:16:12 -06:00
Greg Heartsfield
225c8f762e
improvement: upgrade dependencies; config, tungstenite, tokio
2022-02-26 09:55:12 -06:00
Greg Heartsfield
887fc28ab2
fix: until filters in subscriptions now used
2022-02-26 09:15:45 -06:00
Greg Heartsfield
294d3b99c3
fix: correct imports for test cases
2022-02-26 09:07:07 -06:00
Greg Heartsfield
53990672ae
improvement: move db pool operations closer to query, do not panic on failure
2022-02-23 16:38:16 -06:00
Greg Heartsfield
9c1b21cbfe
improvement: more granular perf logging for SQL queries
2022-02-21 09:03:05 -06:00
Greg Heartsfield
2f63417646
improvement: better logging for connection resets
2022-02-21 08:57:07 -06:00
Greg Heartsfield
3b25160852
fix: abort on connection IO errors
2022-02-21 08:50:46 -06:00
Greg Heartsfield
f8b1fe5035
docs: line up comments with code
2022-02-17 16:18:05 -06:00
Greg Heartsfield
5913b9f87a
feat: send notices when authorization checks fail
2022-02-13 09:35:54 -06:00
Greg Heartsfield
77f35f9f43
feat: server-side pings and disconnects
2022-02-12 16:57:26 -06:00
Greg Heartsfield
9e06cc9482
improvement: better error messages on parse failures
2022-02-12 16:33:29 -06:00
Greg Heartsfield
e66fa4ac42
refactor: remove unnecessary Option wrapping
2022-02-12 16:29:27 -06:00
Greg Heartsfield
99e117f620
improvement: better handling of out-of-protocol messages
2022-02-12 16:26:55 -06:00
Greg Heartsfield
8250e00f05
fix: remove protostream module, and missing NOTICE
2022-02-12 16:22:12 -06:00
Greg Heartsfield
ceaa01e8b4
fix: removed manual nostr stream, so websocket pings work
2022-02-12 16:19:10 -06:00
Greg Heartsfield
e31d0729f2
chore: comment cleanup
2022-02-12 13:49:52 -06:00
Greg Heartsfield
89d96e7ccd
improvement: upgraded database schema to drop legacy tables
...
Database schema is upgraded to version 5. Legacy event and pubkey
tables are dropped, and indexes are added for NIP-05 verification.
2022-02-12 13:47:03 -06:00
Greg Heartsfield
7056aae227
refactor: create schema module
2022-02-12 09:58:42 -06:00
Greg Heartsfield
753df47443
refactor: create utils/hexrange utility modules
2022-02-12 09:29:38 -06:00
Greg Heartsfield
26a0ce2b32
docs: function/struct comments
2022-02-12 09:29:35 -06:00
Greg Heartsfield
fa66a0265e
docs: module headers
2022-02-12 09:29:31 -06:00
Greg Heartsfield
234a8ba0ac
feat: limit event publishing to NIP-05 verified users
...
This adds a new configurable feature to restrict event publishing to
only users with NIP-05 verified metadata. Domains can be whitelisted
or blacklisted. Verification expiration and schedules are
configurable.
This upgrades the database to add a table for tracking verification
records.
2022-02-12 09:29:25 -06:00
Greg Heartsfield
4cc313fa2d
fix: cleanup database connections with same name
...
When a large number of subscriptions is created with identical names,
we do not send a signal over the abandon-read channel. This
eventually leads to resource exhaustion.
2022-01-30 15:14:02 -06:00
Greg Heartsfield
6502f7dcd7
fix: do not panic when validating events with malformed pubkeys
2022-01-29 13:19:34 -06:00
Greg Heartsfield
98c6fa6f39
feat: allow whitelisting of pubkeys for new events
...
This adds a configuration option, `authorization.pubkey_whitelist`
which is an array of pubkeys that are allowed to publish events on
this relay.
2022-01-26 21:39:03 -06:00
Greg Heartsfield
ee0de6f875
improvement: clearer and less verbose database logging
2022-01-25 21:42:43 -06:00
Greg Heartsfield
a72eaec3b8
fix: never display hidden events
2022-01-25 20:48:46 -06:00
Greg Heartsfield
f1206e76f2
feat: database reader connection pooling
...
Added connection pooling for queries, as well as basic configuration
options for min/max connections.
2022-01-25 20:39:24 -06:00
Greg Heartsfield
af453548ee
feat: allow author and event id prefix search
...
This is an experimental non-NIP feature that allows a subscription
filter to include a prefix for authors and events.
2022-01-25 18:23:08 -06:00
Greg Heartsfield
2d28a95ff7
feat: allow arbitrary tag queries
...
This is an experimental feature, outside of any NIP, that demonstrates
generic tag queries.
Instead of limiting subscription filters to just querying only "e" or
"p" tags (via `#e` or `#p` attributes), any tag can be queried.
As an example, consider an event which uses a tag "url". With this
modification, a subscription filter could add a top-level field
"#url", with an array of strings as the key. Exact matches would be
returned.
A NIP is forthcoming to formalize this.
2022-01-22 21:29:15 -06:00
Raj
179928378e
refactor: add strictly typed tags
...
* Add custom error variant
This can be useful to propagate errors not conforming to available
variants. Also to convert other errors in `crate::Error` without having
explicit conversion defined, with `error.to_string()`
* Implement `Tag` and define protocol serialization
A Tag structure have been implemented with dedicated field types. Then
custom serde serialization is derived to map the structure to current
protocol json array as per NIP01.
This adds compile and run time type checking to always ensure wrong
string data are never stored or processed. With strict typed fields and
custom serde derivation this checks can be done at time of serialization,
saving work for internal handling of the actual data.
tests for possible data violations are added, and gives good example of
kind of errors it will through for different cases.
* Use String for URL
2022-01-19 07:42:58 -06:00
Greg Heartsfield
81e4e2b892
feat: add supported NIPs (2, 11) to relay info
2022-01-16 08:37:21 -06:00
Greg Heartsfield
6f166433b5
fix: test failures
2022-01-16 08:36:52 -06:00
Greg Heartsfield
030b64de62
feat: replace email with contact field in relay info.
...
This finalizes the NIP-11 spec implementation.
Fixes https://todo.sr.ht/~gheartsfield/nostr-rs-relay/21 .
2022-01-16 08:34:19 -06:00
Greg Heartsfield
a3124ccea4
improvement: better sql error handling
2022-01-15 09:42:53 -06:00
Greg Heartsfield
4e51e61d16
improvement: display rate limit messages max once per sec
2022-01-15 09:42:17 -06:00
Raj
5c8390bbe0
fix: fix some test failures
2022-01-14 14:27:12 -06:00
Greg Heartsfield
da7968efef
fix: restore working websocket message size configuration options
2022-01-05 17:41:12 -05:00
Greg Heartsfield
7037555516
improvement: add indexed tag queries
2022-01-05 17:33:53 -05:00
Greg Heartsfield
19ed990c57
refactor: fix clippy errors for relay info response
2022-01-05 10:10:44 -05:00
