nostr/nostr-rs-relay
1
0
Fork 0
mirror of https://github.com/scsibug/nostr-rs-relay.git synced 2024-11-14 23:19:07 -05:00
Code Issues Packages Projects Releases Wiki Activity
448 Commits 4 Branches 54 Tags 2.5 MiB
0.8.5
Commit Graph

305 Commits

Author SHA1 Message Date
Greg Heartsfield
ce0e00ffb3 feat: log reader DB pool stats every minute 2022-12-25 10:47:32 -06:00
Greg Heartsfield
3e4ae4aeec feat: cache prepared statements and trace expanded SQL queries 2022-12-25 10:47:32 -06:00
Greg Heartsfield
c6a8807485 improvement: send error on empty-string prefix author/id searches 2022-12-25 10:47:32 -06:00
Greg Heartsfield
8137b6211c refactor: clippy suggestions 2022-12-24 10:29:47 -06:00
Greg Heartsfield
e5074f2e46 feat(NIP-28): replaceable kind 41 channel metadata events 2022-12-24 10:14:43 -06:00
Blake Jakopovic
4fd7643907 feat: change pub(crate) to pub for use as a library 2022-12-23 07:14:58 -06:00
benthecarman
e08647867c refactor: remove code duplication for simple_event 2022-12-23 06:39:50 -06:00
Greg Heartsfield
4f1a912f36 feat: log origin header from websocket requests 
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/33
 2022-12-22 16:55:53 -06:00
Greg Heartsfield
25480e837f fix: do not block writers for more than 1 second during checkpoints 2022-12-22 16:10:49 -06:00
Greg Heartsfield
b80b54cd9d improvement: reduce logging, especially for database pool size 2022-12-22 15:47:33 -06:00
Greg Heartsfield
8ea732cbe5 feat: perform regular database maintenance (60sec), without blocking main writer thread 2022-12-22 15:16:21 -06:00
Greg Heartsfield
0f68c4e5c2 refactor: formatting 2022-12-22 15:15:45 -06:00
Greg Heartsfield
f411aa6fc2 fix: do not re-verify NIP-05 entries where metadata was deleted 2022-12-22 13:01:48 -06:00
Greg Heartsfield
d31bbda087 improvement: reduce lifetime of database connections 2022-12-22 13:01:12 -06:00
Greg Heartsfield
5917bc53b2 improvement: run maintenance every 60 seconds instead of by event count 2022-12-22 11:40:17 -06:00
Greg Heartsfield
91177c61a1 improvement: log reason for new event creation from nip05 2022-12-22 10:48:30 -06:00
Greg Heartsfield
53c2a8051c improvement: reduce logging 2022-12-22 10:29:27 -06:00
Greg Heartsfield
168cf513ac feat: perform full checkpoints and truncate WAL every 2k events 2022-12-22 10:11:05 -06:00
Greg Heartsfield
ea204761c9 fix: do not show slow queries more than once per sub 2022-12-20 15:41:50 -06:00
Greg Heartsfield
c270ae1434 improvement: reduce event count for db writer pauses 2022-12-20 15:25:24 -06:00
Greg Heartsfield
64bd983cb6 perf: every 5000 persisted events, pause for 500ms for backups 
I have observed backups running for a very long time under heavy load,
this introduces some artificial delay to give the online backup enough
time to make progress.
 2022-12-20 15:05:04 -06:00
Greg Heartsfield
1c153bc784 perf: shed DB query load when queue gets large 2022-12-20 13:23:21 -06:00
Greg Heartsfield
dc11d9a619 improvement: explicitly rollback transaction on duplicate event 2022-12-20 13:23:04 -06:00
Greg Heartsfield
cd1557787b improvement: log write pool 2022-12-20 13:21:57 -06:00
Greg Heartsfield
86bb7aeb9a improvement: function to check pool capacity 2022-12-20 10:07:01 -06:00
Greg Heartsfield
2cfd384339 perf: drop db handles that are not quickly read 2022-12-19 00:18:39 -06:00
Greg Heartsfield
8c013107f9 perf: increase upper bound for sqlite mmap 2022-12-18 23:19:43 -06:00
Greg Heartsfield
64a4466d30 perf: backing down on max_blocking_threads 2022-12-18 23:14:41 -06:00
Greg Heartsfield
1596c23eb4 perf: increase blocking threads now that contention is reduced 2022-12-18 22:46:32 -06:00
Greg Heartsfield
129badd4e1 perf: reduce per thread mmap allocation for DB 2022-12-18 22:45:32 -06:00
Greg Heartsfield
6f7c080180 improvement: reduce number of writer blocking threads from 4->2 2022-12-18 22:32:31 -06:00
Greg Heartsfield
af92561ef6 perf: remove shared cache mode (experiment) 2022-12-18 22:15:50 -06:00
Greg Heartsfield
d833a3e40d perf: reduce logging 2022-12-18 22:11:46 -06:00
Greg Heartsfield
cf144d503d perf: reduce logging for slow queries 2022-12-18 20:47:11 -06:00
Greg Heartsfield
88ac31b549 perf: increase channel size for DB communication 2022-12-18 13:44:28 -06:00
Greg Heartsfield
677b7d39e9 improvement: log slow requests that return zero results 2022-12-18 13:42:31 -06:00
Greg Heartsfield
b24d2f9aaa perf: set default blocking threads to lower value 2022-12-18 12:20:57 -06:00
Greg Heartsfield
d10348f7e1 feat: configurable blocking threads 2022-12-18 09:14:04 -06:00
Greg Heartsfield
8598e443d8 wip: add configuration for future feature (client concurrent db limits) 2022-12-17 23:19:48 -06:00
Greg Heartsfield
43222d44e5 feat: perform optimization after seeing many events 2022-12-17 23:18:54 -06:00
Greg Heartsfield
7c1516c4fb perf: add index for tags 2022-12-17 23:17:53 -06:00
Greg Heartsfield
0c72053a49 perf: increase mmap size to 1GB 2022-12-17 23:17:16 -06:00
Greg Heartsfield
3f32ff67ab improvement: minor logging 2022-12-17 23:11:14 -06:00
Greg Heartsfield
0b9778d6ca refactor: simplify tracking of subscriptions 2022-12-17 20:46:58 -06:00
Greg Heartsfield
cc06167e06 perf: add composite index for tag table 2022-12-17 12:01:20 -06:00
Greg Heartsfield
b6e33f044f improvement: limit db connection max lifetime 2022-12-17 10:47:35 -06:00
Greg Heartsfield
0d8d39ad22 feat: add rate limiting setting for subscription creation 2022-12-17 09:27:29 -06:00
Greg Heartsfield
3c880b2f49 perf: pull distinct to outermost SQL 2022-12-17 07:49:28 -06:00
Greg Heartsfield
7a4c9266ec improvement: make hexsearch structs sortable 2022-12-17 07:49:05 -06:00
Greg Heartsfield
7ca9c864f2 improvement: DB pool logging shows used connections directly 2022-12-16 17:01:49 -06:00
Greg Heartsfield
838aafd079 improvement: consistent log messages for client/sub ids 2022-12-16 15:22:27 -06:00
Greg Heartsfield
e554b10ac2 improvement: tweak sub/sql logging for slow queries 2022-12-16 14:55:45 -06:00
Greg Heartsfield
b0bfaa48fc improvement: ignore duplicate REQ messages 2022-12-16 14:37:02 -06:00
Greg Heartsfield
2e9b1b6ba7 docs: comment reason for force_no_match 2022-12-16 14:35:21 -06:00
Greg Heartsfield
f9695bd0a9 fix: db schema version updates correctly for v9 2022-12-16 10:01:49 -06:00
Greg Heartsfield
7c4bf5cc8f fix: run db migration for v9 2022-12-16 08:21:00 -06:00
Greg Heartsfield
e2de162931 feat: only show SQL in logs for slow queries unless tracing 2022-12-16 08:17:39 -06:00
Greg Heartsfield
4f606615eb perf: indexing improvement 2022-12-16 08:16:49 -06:00
Greg Heartsfield
c48e45686d perf: schema updates for better event indexing 2022-12-15 08:48:35 -06:00
Greg Heartsfield
bbe359364a refactor: clippy warnings 2022-12-15 08:43:36 -06:00
Greg Heartsfield
9e9c494367 perf: significant query speedup when using kinds. 
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/51
 2022-12-14 21:04:49 -06:00
Greg Heartsfield
5fa24bc9f1 fix: send EOSE when ids list is empty in subscriptions 
Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/50
 2022-11-19 10:35:00 -06:00
Greg Heartsfield
4de7490d97 fix: send EOSE when authors list is empty in subscriptions 
Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/49
 2022-11-19 10:00:38 -06:00
Greg Heartsfield
110500bb46 feat(NIP-20): advertise support for NIP-20 in relay info/readme 2022-11-12 09:22:43 -06:00
Greg Heartsfield
83f6b11de7 refactor: clippy fix 2022-11-12 09:22:24 -06:00
William Casarin
6d1244434b feat(NIP-20): improve invalid event error messages 
Instead of returning a NOTICE for invalid events, return a `OK false`
command result with a reason as to why the event is invalid.
 2022-11-12 09:13:22 -06:00
William Casarin
5a91419d34 feat(NIP-20): send command results to clients 
When submitting events to relays, clients currently have no way to know
if an event was successfully committed to the database. This NIP
introduces the concept of command results which are like NOTICE's except
provide more information about if an event was accepted or rejected.

A command result is a JSON object with the following structure that is
returned when an event is successfully saved to the database or
rejected:

	["OK", <event_id>, <true|false>, <message>]

nip20: https://github.com/nostr-protocol/nips/pull/62
 2022-11-12 09:12:35 -06:00
William Casarin
7adc5c9af7 perf: dont create intermediate vecs when matching subs 
Avoid creating intermediate vectors when matching subscriptions. We can
just iterate over the hashmap directly.
 2022-11-09 07:30:43 -06:00
Greg Heartsfield
9dd4571bee refactor: reduce level of some common DB logs 2022-11-06 13:49:32 -06:00
Greg Heartsfield
9db5a26b9c refactor: more consistent logging messages 2022-11-05 16:11:20 -05:00
Greg Heartsfield
ac345b5744 refactor: do not quote server-generated client id in logs 2022-11-05 15:59:39 -05:00
Greg Heartsfield
5a8860bb09 feat: log user-agent if present 2022-11-05 10:29:25 -05:00
Greg Heartsfield
11e43eccf9 refactor: add unit to ping_interval config 2022-11-05 07:42:08 -05:00
William Casarin
50577b2dfa feat: add network.ping_interval setting 
Add a ping interval setting that allows you to customize the websocket
ping interval. The default of 5 minutes may be too high for some proxy
servers that disconnect connections that are held open for too long.
 2022-11-05 07:40:28 -05:00
William Casarin
a6cb6f8486 refactor: rename get_header_remote_ip -> get_header_string 
This function has nothing to do with remote ips!
 2022-11-05 07:37:18 -05:00
Greg Heartsfield
ae5bf98d87 feat: retrieve client IP from header in config.toml 
If the config.toml has defined a HTTP header to look for a remote IP,
that will be logged.  Otherwise, the socket address IP will be used.

closes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/47
 2022-11-04 18:05:01 -05:00
William Casarin
1cf9d719f0 feat: look for proxied ip headers 
This enables support for using the proxied IP from cloudflare. The damus
relay is behind cloudflare, so to get accurate remote ip logging we need
to look at the headers instead of the socket address.

Signed-off-by: William Casarin <jb55@jb55.com>
 2022-11-04 17:09:28 -05:00
William Casarin
311f4b5283 refactor: switch new connections to debug log 
These are pretty spammy on busy relays. I've been using the info log to
monitor spam attacks, and these are the least useful info log.

Leave the "stopping connection" log because it at least provides useful
sent/received information.

Signed-off-by: William Casarin <jb55@jb55.com>
 2022-11-04 07:59:53 -05:00
Greg Heartsfield
14b5a51e3a fix: log ephemeral events after send 2022-11-04 07:55:38 -05:00
Greg Heartsfield
8ecce3f566 feat: show client IP in logs 2022-11-02 18:33:44 -05:00
Greg Heartsfield
72f8a1aa5c feat(NIP-26): allow searches for delegated public keys 
Implements core NIP-26 delegated event functionality.  Events can
include a `delegation` tag that provides a signature and restrictions
on which events can be delegated.

Notable points on the implementation so far:

* Schema has been upgraded to include an index and new column.
* Basic rune parsing/evaluation to implement the example event in the
  NIP, but no more.
* No special logic for deletion.
* No migration logic for determining delegated authors for
  already-stored events.
 2022-10-16 15:25:06 -05:00
Greg Heartsfield
3e8adf978f refactor: move db migrations into isolated functions 2022-10-09 08:54:03 -05:00
Greg Heartsfield
2af5f9fbe8 fix: correct schema upgrade logic (and refactor) 
Schema upgrades were buggy from 4->5 (the v5 would be skipped).  This
change also refactors the logic slightly so that future additions can
be clearer (no need to have if and else-if combinations).
 2022-10-09 08:24:01 -05:00
Greg Heartsfield
2739e49362 fix: correct future schema version detection 2022-10-08 13:15:48 -05:00
Greg Heartsfield
f9693f7ac3 fix(NIP-9): hide events received after their deletions 
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/40
 2022-10-08 12:12:41 -05:00
Greg Heartsfield
8a63d88b0b fix: prevent deletion of deletion events 2022-10-08 08:02:16 -05:00
Greg Heartsfield
92da9d71f8 feat: handle and log TERM signals 2022-09-28 07:20:31 -05:00
Greg Heartsfield
6633f8b472 feat: replace logging with tracing 2022-09-28 07:19:59 -05:00
Greg Heartsfield
93dfed0a87 refactor: misc clippy suggestions 2022-09-24 19:28:02 -05:00
Greg Heartsfield
bef7ca7e27 refactor: misc clippy suggestions 2022-09-24 09:19:16 -05:00
Greg Heartsfield
a98708ba47 refactor: misc clippy suggestions 2022-09-24 09:01:09 -05:00
Greg Heartsfield
ccf9b8d47b refactor: remove unnecessary return types 2022-09-24 08:39:41 -05:00
Greg Heartsfield
8fa58de49a refactor: clippy suggestions 2022-09-24 08:30:22 -05:00
Greg Heartsfield
4fa8616c73 feat: enable use of tokio-console with diagnostics.tracing setting 
View real-time tokio diagnostics by setting the configuration option
"diagnostics.tracing" to true.
 2022-09-11 12:44:45 -05:00
Greg Heartsfield
74802522c2 improvement: do not create NIP-05 thread if feature is disabled 2022-09-11 11:01:36 -05:00
Greg Heartsfield
9ce5057af8 improvement: better log formatting 2022-09-11 10:22:01 -05:00
Greg Heartsfield
78da92ccca feat: advertise support for NIP-09 and NIP-12 in relay info 
NIP-01 prefix search, and NIP-12 generic tags are no longer marked as
experimental.

NIP-11 relay info advertises NIP-09 event deletion and NIP-12 generic
tag search support.
 2022-09-10 20:45:09 -05:00
Greg Heartsfield
72f1c19b21 feat(NIP-22): advertise support for event created_at limits 
The `reject_future_limits` option can now be disabled, and is by
default.

NIP-11 advertises support for created_at limits.

The message for future-dated events has been modified, to be closer to
the recommended example in the NIP.
 2022-09-10 20:40:10 -05:00
Greg Heartsfield
08b011ad07 feat: ensure that WAL is truncated after checkpoint 2022-09-10 19:18:57 -05:00
Greg Heartsfield
2b03f11e5e refactor: remove global/singleton settings object 2022-09-06 06:12:07 -05:00
Greg Heartsfield
e48bae10e6 feat: support in-memory SQLite database 2022-09-06 06:06:01 -05:00
Greg Heartsfield
8774416b92 refactor: move nostr server into library 2022-09-06 05:56:04 -05:00
Greg Heartsfield
1b9f364e15 chore: rustfmt 2022-09-02 12:38:31 -05:00
Greg Heartsfield
a754477a02 improvement: misc refactorings (clippy) 2022-09-02 12:26:00 -05:00
Greg Heartsfield
a843eaa939 improvement: db.rs from clippy 2022-09-02 10:30:51 -05:00
Greg Heartsfield
03a130b0b8 improvement: simplify config builder (clippy) 2022-09-02 10:18:16 -05:00
slaninas
77892b2064 fix: syntax error 2022-08-22 05:12:52 -07:00
Greg Heartsfield
4fe6191aa3 chore: formatting 2022-08-21 09:51:34 -07:00
Greg Heartsfield
79a982e3ef improvement: send NOTICE for too-large messages 2022-08-21 09:28:31 -07:00
Greg Heartsfield
01d81db617 improvement: log client id for subscription removal 2022-08-21 09:11:38 -07:00
Greg Heartsfield
e6fef37d4e chore: rustfmt 2022-08-21 09:10:19 -07:00
Greg Heartsfield
035cf34673
fix(NIP-12): correctly search for mixed-case hex-like tags 
Only lowercase and even-length tag values are stored as binary BLOBs.
Previously there was an error which search results from being returned
if the tag value was mixed-case and could be interpreted as hex.

A new database migration has been created to repair the `tag` table
for existing relays.

fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/37
 2022-08-17 16:34:11 -07:00
Greg Heartsfield
be8170342e
fix(NIP-12): multi-tag searches returns correct results 
Logic of generated SQL was incorrect, causing multiple tag searches
(as defined in NIP-12) to produce no results.

fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/36
 2022-08-11 22:16:10 -07:00
Greg Heartsfield
0a3b15f41f
fix(NIP-11): Add CORS header and content type for main page 2022-08-11 19:33:17 -07:00
Greg Heartsfield
5058d98ad6
fix(NIP-12): only allow single-char tag filters 2022-08-07 10:15:36 -05:00
Greg Heartsfield
1c14adc766 fix(NIP-01): allow limits on a per-filter basis 
The original implementation of subscription limit applied to the
entire query, instead of the specific filter.  Now, each filter gets
its own query limit.  When a limit is applied, the most recent N
events will be returned, otherwise the default is to return the
earliest events (in order), for all matching events.
 2022-07-04 17:25:32 -05:00
Semisol
168cfc3b26 feat(NIP-16): Implement NIP16 
NIP16 introduces a replaceable and ephemeral event range:
[10000..20000) for replaceable and [20000..30000) for
ephemeral.
 2022-05-30 21:43:06 -05:00
Semisol
a36ad378f6 feat(NIP-15): Implement NIP15 
NIP15 sends an EOSE notice to clients after all stored events are sent
to allow loading indicators and other use cases.
 2022-05-30 21:43:00 -05:00
Greg Heartsfield
9ed3391b46 fix(NIP-09): correct WHERE clause for event deletion 2022-05-10 16:50:52 -05:00
William Casarin
4ad483090e feat(NIP-01): Implement limit 
This was quickly sneaked in by fiatjaf per my request[0], it makes many
queries more efficient and allows for paging when combined with until.

It is a bit weird to have multiple limits on each filter... for now we
just choose any or the last limit seen.

[0]: a4aea5337f

Signed-off-by: William Casarin <jb55@jb55.com>
 2022-05-10 16:47:56 -05:00
Greg Heartsfield
597749890e improvement: remove unnecessary event logging 2022-02-27 19:30:48 -06:00
Greg Heartsfield
1d499cf12b feat: handle NIP-09 for deletion events 2022-02-27 11:35:23 -06:00
Greg Heartsfield
ed3a6b9692 refactor: simplify NOTICE messages 2022-02-26 17:34:58 -06:00
Greg Heartsfield
414e83f696 refactor: import cleanup for config 2022-02-26 11:16:12 -06:00
Greg Heartsfield
225c8f762e improvement: upgrade dependencies; config, tungstenite, tokio 2022-02-26 09:55:12 -06:00
Greg Heartsfield
887fc28ab2 fix: until filters in subscriptions now used 2022-02-26 09:15:45 -06:00
Greg Heartsfield
294d3b99c3 fix: correct imports for test cases 2022-02-26 09:07:07 -06:00
Greg Heartsfield
53990672ae improvement: move db pool operations closer to query, do not panic on failure 2022-02-23 16:38:16 -06:00
Greg Heartsfield
9c1b21cbfe improvement: more granular perf logging for SQL queries 2022-02-21 09:03:05 -06:00
Greg Heartsfield
2f63417646 improvement: better logging for connection resets 2022-02-21 08:57:07 -06:00
Greg Heartsfield
3b25160852 fix: abort on connection IO errors 2022-02-21 08:50:46 -06:00
Greg Heartsfield
f8b1fe5035 docs: line up comments with code 2022-02-17 16:18:05 -06:00
Greg Heartsfield
5913b9f87a feat: send notices when authorization checks fail 2022-02-13 09:35:54 -06:00
Greg Heartsfield
77f35f9f43 feat: server-side pings and disconnects 2022-02-12 16:57:26 -06:00
Greg Heartsfield
9e06cc9482 improvement: better error messages on parse failures 2022-02-12 16:33:29 -06:00
Greg Heartsfield
e66fa4ac42 refactor: remove unnecessary Option wrapping 2022-02-12 16:29:27 -06:00
Greg Heartsfield
99e117f620 improvement: better handling of out-of-protocol messages 2022-02-12 16:26:55 -06:00
Greg Heartsfield
8250e00f05 fix: remove protostream module, and missing NOTICE 2022-02-12 16:22:12 -06:00
Greg Heartsfield
ceaa01e8b4 fix: removed manual nostr stream, so websocket pings work 2022-02-12 16:19:10 -06:00
Greg Heartsfield
e31d0729f2 chore: comment cleanup 2022-02-12 13:49:52 -06:00
Greg Heartsfield
89d96e7ccd improvement: upgraded database schema to drop legacy tables 
Database schema is upgraded to version 5.  Legacy event and pubkey
tables are dropped, and indexes are added for NIP-05 verification.
 2022-02-12 13:47:03 -06:00
Greg Heartsfield
7056aae227 refactor: create schema module 2022-02-12 09:58:42 -06:00
Greg Heartsfield
753df47443 refactor: create utils/hexrange utility modules 2022-02-12 09:29:38 -06:00
Greg Heartsfield
26a0ce2b32 docs: function/struct comments 2022-02-12 09:29:35 -06:00
Greg Heartsfield
fa66a0265e docs: module headers 2022-02-12 09:29:31 -06:00
Greg Heartsfield
234a8ba0ac feat: limit event publishing to NIP-05 verified users 
This adds a new configurable feature to restrict event publishing to
only users with NIP-05 verified metadata.  Domains can be whitelisted
or blacklisted.  Verification expiration and schedules are
configurable.

This upgrades the database to add a table for tracking verification
records.
 2022-02-12 09:29:25 -06:00
Greg Heartsfield
4cc313fa2d fix: cleanup database connections with same name 
When a large number of subscriptions is created with identical names,
we do not send a signal over the abandon-read channel.  This
eventually leads to resource exhaustion.
 2022-01-30 15:14:02 -06:00
Greg Heartsfield
6502f7dcd7 fix: do not panic when validating events with malformed pubkeys 2022-01-29 13:19:34 -06:00
Greg Heartsfield
98c6fa6f39 feat: allow whitelisting of pubkeys for new events 
This adds a configuration option, `authorization.pubkey_whitelist`
which is an array of pubkeys that are allowed to publish events on
this relay.
 2022-01-26 21:39:03 -06:00
Greg Heartsfield
ee0de6f875 improvement: clearer and less verbose database logging 2022-01-25 21:42:43 -06:00