From 8d6d58871542dfcbb3f9d5d610a13a1e17358e58 Mon Sep 17 00:00:00 2001 From: kieran Date: Tue, 14 May 2024 13:59:59 +0100 Subject: [PATCH 1/4] nip96 list uploads --- 96.md | 108 ++++++++++++++++++++++++++++++++++++---------------------- 1 file changed, 68 insertions(+), 40 deletions(-) diff --git a/96.md b/96.md index f7d901fd..4203b63e 100644 --- a/96.md +++ b/96.md @@ -84,46 +84,43 @@ it must use the "api_url" field instead. See https://github.com/aljazceru/awesome-nostr#nip-96-file-storage-servers. + +## Auth + +When indicated, `clients` must add an [NIP-98](98.md) `Authorization` header (**optionally** with the encoded `payload` tag set to the base64-encoded 256-bit SHA-256 hash of the file - not the hash of the whole request body). + ## Upload -A file can be uploaded one at a time to `https://your-file-server.example/custom-api-path` (route from `https://your-file-server.example/.well-known/nostr/nip96.json` "api_url" field) as `multipart/form-data` content type using `POST` method with the file object set to the `file` form data field. +`POST $api_url` as `multipart/form-data`. -`Clients` must add an [NIP-98](98.md) `Authorization` header (**optionally** with the encoded `payload` tag set to the base64-encoded 256-bit SHA-256 hash of the file - not the hash of the whole request body). -If using an html form, use an `Authorization` form data field instead. +**AUTH required** -These following **optional** form data fields MAY be used by `servers` and SHOULD be sent by `clients`: -- `expiration`: string of the UNIX timestamp in seconds. Empty string if file should be stored forever. The server isn't required to honor this; -- `size`: string of the file byte size. This is just a value the server can use to reject early if the file size exceeds the server limits; -- `alt`: (recommended) strict description text for visibility-impaired users; -- `caption`: loose description; -- `media_type`: "avatar" or "banner". Informs the server if the file will be used as an avatar or banner. If absent, the server will interpret it as a normal upload, without special treatment; +List of form fields: +- `file`: **REQUIRED** the file to upload +- `caption`: **RECOMMENDED** loose description; +- `expiration`: UNIX timestamp in seconds. Empty string if file should be stored forever. The server isn't required to honor this. +- `size`: File byte size. This is just a value the server can use to reject early if the file size exceeds the server limits. +- `alt`: **RECOMMENDED** strict description text for visibility-impaired users. +- `media_type`: "avatar" or "banner". Informs the server if the file will be used as an avatar or banner. If absent, the server will interpret it as a normal upload, without special treatment. - `content_type`: mime type such as "image/jpeg". This is just a value the server can use to reject early if the mime type isn't supported. - Others custom form data fields may be used depending on specific `server` support. The `server` isn't required to store any metadata sent by `clients`. -Note for `clients`: if using an HTML form, it is important for the `file` form field to be the **last** one, or be re-ordered right before sending or be appended as the last field of XHR2's FormData object. - The `filename` embedded in the file may not be honored by the `server`, which could internally store just the SHA-256 hash value as the file name, ignoring extra metadata. -The hash is enough to uniquely identify a file, that's why it will be used on the "download" and "delete" routes. +The hash is enough to uniquely identify a file, that's why it will be used on the `download` and `delete` routes. -The `server` MUST link the user's `pubkey` string (which is embedded in the decoded header value) as the owner of the file so to later allow them to delete the file. -Note that if a file with the same hash of a previously received file (so the same file) is uploaded by another user, the server doesn't need to store the new file. -It should just add the new user's `pubkey` to the list of the owners of the already stored file with said hash (if it wants to save space by keeping just one copy of the same file, because multiple uploads of the same file results in the same file hash). +The `server` MUST link the user's `pubkey` string as the owner of the file so to later allow them to delete the file. -The `server` MAY also store the `Authorization` header/field value (decoded or not) for accountability purpose as this proves that the user with the unique pubkey did ask for the upload of the file with a specific hash. However, storing the pubkey is sufficient to establish ownership. +### Response codes -The `server` MUST reject with 413 Payload Too Large if file size exceeds limits. - -The `server` MUST reject with 400 Bad Request status if some fields are invalid. - -The `server` MUST reply to the upload with 200 OK status if the `payload` tag value contains an already used SHA-256 hash (if file is already owned by the same pubkey) or reject the upload with 403 Forbidden status if it isn't the same of the received file. - -The `server` MAY reject the upload with 402 Payment Required status if the user has a pending payment (Payment flow is not strictly required. Server owners decide if the storage is free or not. Monetization schemes may be added later to correlated NIPs.). - -On successful uploads the `server` MUST reply with **201 Created** HTTP status code or **202 Accepted** if a `processing_url` field is added -to the response so that the `client` can follow the processing status (see [Delayed Processing](#delayed-processing) section). +- `200 OK`: File upload exists, but is successful (Existing hash) +- `201 Created`: File upload successful (New hash) +- `202 Accepted`: File upload is awaiting processing, see [Delayed Processing](#delayed-processing) section +- `413 Payload Too Large`: File size exceeds limit +- `400 Bad Request`: Form data is invalid or not supported. +- `403 Forbidden`: User is not allowed to upload or the uploaded file hash didnt match the hash included in the `Authorization` header `payload` tag. +- `402 Payment Required`: Payment is required by the server, **this flow is undefined**. The upload response is a json object as follows: @@ -179,11 +176,13 @@ The upload response is a json object as follows: Note that if the server didn't apply any transformation to the received file, both `nip94_event.tags.*.ox` and `nip94_event.tags.*.x` fields will have the same value. The server MUST link the saved file to the SHA-256 hash of the **original** file before any server transformations (the `nip94_event.tags.*.ox` tag value). The **original** file's SHA-256 hash will be used to identify the saved file when downloading or deleting it. -`Clients` may upload the same file to one or many `servers`. +`clients` may upload the same file to one or many `servers`. After successful upload, the `client` may optionally generate and send to any set of nostr `relays` a [NIP-94](94.md) event by including the missing fields. Alternatively, instead of using NIP-94, the `client` can share or embed on a nostr note just the above url. +`clients` may also use the tags from the `nip94_event` to construct an `imeta` tag + ### Delayed Processing Sometimes the server may want to place the uploaded file in a processing queue for deferred file processing. @@ -219,7 +218,7 @@ However, for all file actions, such as download and deletion, the **original** f ## Download -`Servers` must make available the route `https://your-file-server.example/custom-api-path/(.ext)` (route taken from `https://your-file-server.example/.well-known/nostr/nip96.json` "api_url" or "download_url" field) with `GET` method for file download. +`GET $api_url/(.ext)` The primary file download url informed at the upload's response field `nip94_event.tags.*.url` can be that or not (it can be any non-standard url the server wants). @@ -227,17 +226,17 @@ If not, the server still MUST also respond to downloads at the standard url mentioned on the previous paragraph, to make it possible for a client to try downloading a file on any NIP-96 compatible server by knowing just the SHA-256 file hash. -Note that the "\" part is from the **original** file, **not** from the **transformed** file if the uploaded file went through any server transformation. +Note that the "\" part is from the **original** file, **not** from the **transformed** file if the uploaded file went through any server transformation. Supporting ".ext", meaning "file extension", is required for `servers`. It is optional, although recommended, for `clients` to append it to the path. When present it may be used by `servers` to know which `Content-Type` header to send (e.g.: "Content-Type": "image/png" for ".png" extension). The file extension may be absent because the hash is the only needed string to uniquely identify a file. -Example: `https://your-file-server.example/custom-api-path/719171db19525d9d08dd69cb716a18158a249b7b3b3ec4bbdec5698dca104b7b.png` +Example: `$api_url/719171db19525d9d08dd69cb716a18158a249b7b3b3ec4bbdec5698dca104b7b.png` ### Media Transformations -`Servers` may respond to some media transformation query parameters and ignore those they don't support by serving +`servers` may respond to some media transformation query parameters and ignore those they don't support by serving the original media file without transformations. #### Image Transformations @@ -245,23 +244,23 @@ the original media file without transformations. ##### Resizing Upon upload, `servers` may create resized image variants, such as thumbnails, respecting the original aspect ratio. -`Clients` may use the `w` query parameter to request an image version with the desired pixel width. -`Servers` can then serve the variant with the closest width to the parameter value +`clients` may use the `w` query parameter to request an image version with the desired pixel width. +`servers` can then serve the variant with the closest width to the parameter value or an image variant generated on the fly. -Example: `https://your-file-server.example/custom-api-path/.png?w=32` +Example: `$api_url/.png?w=32` ## Deletion -`Servers` must make available the route `https://deletion.domain/deletion-path/(.ext)` (route taken from `https://your-file-server.example/.well-known/nostr/nip96.json` "api_url" field) with `DELETE` method for file deletion. +`DELETE $api_url/(.ext)` -Note that the "\" part is from the **original** file, **not** from the **transformed** file if the uploaded file went through any server transformation. +**AUTH required** + +Note that the `/` part is from the **original** file, **not** from the **transformed** file if the uploaded file went through any server transformation. The extension is optional as the file hash is the only needed file identification. -`Clients` should send a `DELETE` request to the server deletion route in the above format. It must include a NIP-98 `Authorization` header. - -The `server` should reject deletes from users other than the original uploader. The `pubkey` encoded on the header value identifies the user. +The `server` should reject deletes from users other than the original uploader with the appropriate http response code (403 Forbidden). It should be noted that more than one user may have uploaded the same file (with the same hash). In this case, a delete must not really delete the file but just remove the user's `pubkey` from the file owners list (considering the server keeps just one copy of the same file, because multiple uploads of the same file results in the same file hash). @@ -275,6 +274,35 @@ The successful response is a 200 OK one with just basic JSON fields: } ``` +## Listing files + +`GET $api_url` + +**AUTH required** + +Returns a list of files linked to the authenticated users pubkey. + +Example Response: +```js +[ + { + "id": "", + "nip94_event": {...}, + "expires": 1715691139, // unix timestamp + "size": 123456, + "alt": "a meme that makes you laugh", + "caption": "haha funny meme" + }, + ... +] +``` + +`` is the **original hash**, ie. `ox` + +`nip94_event` is the same as in the upload result. + +`alt` / `caption` are optional. + ## Selecting a Server Note: HTTP File Storage Server developers may skip this section. This is meant for client developers. From bd9c7a1b8e372b3c8e518861b1f4ea5c92ef1888 Mon Sep 17 00:00:00 2001 From: kieran Date: Mon, 20 May 2024 21:38:36 +0100 Subject: [PATCH 2/4] add pagination / drop duplicate fields --- 96.md | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/96.md b/96.md index 4203b63e..e882870e 100644 --- a/96.md +++ b/96.md @@ -1,8 +1,6 @@ -NIP-96 -====== +# NIP-96 -HTTP File Storage Integration ------------------------------ +## HTTP File Storage Integration `draft` `optional` @@ -84,8 +82,7 @@ it must use the "api_url" field instead. See https://github.com/aljazceru/awesome-nostr#nip-96-file-storage-servers. - -## Auth +## Auth When indicated, `clients` must add an [NIP-98](98.md) `Authorization` header (**optionally** with the encoded `payload` tag set to the base64-encoded 256-bit SHA-256 hash of the file - not the hash of the whole request body). @@ -96,6 +93,7 @@ When indicated, `clients` must add an [NIP-98](98.md) `Authorization` header (** **AUTH required** List of form fields: + - `file`: **REQUIRED** the file to upload - `caption`: **RECOMMENDED** loose description; - `expiration`: UNIX timestamp in seconds. Empty string if file should be stored forever. The server isn't required to honor this. @@ -276,22 +274,27 @@ The successful response is a 200 OK one with just basic JSON fields: ## Listing files -`GET $api_url` +`GET $api_url?page=x&count=y` **AUTH required** Returns a list of files linked to the authenticated users pubkey. Example Response: + ```js [ - { + { "id": "", - "nip94_event": {...}, - "expires": 1715691139, // unix timestamp - "size": 123456, - "alt": "a meme that makes you laugh", - "caption": "haha funny meme" + "nip94_event": { + "tags": [ + ["size", "123456"], + ["alt", "a meme that makes you laugh"], + ["expiration", "1715691139"] + // ...other metadata + ] + "content": "haha funny meme" // caption + } }, ... ] @@ -301,7 +304,10 @@ Example Response: `nip94_event` is the same as in the upload result. -`alt` / `caption` are optional. +### Query args + +- `page` page number (`offset=page*count`) +- `count` number of items per page ## Selecting a Server From 17593a41ab7ca51305db07cbfe1866f88e790206 Mon Sep 17 00:00:00 2001 From: Kieran Date: Mon, 27 May 2024 14:52:49 +0100 Subject: [PATCH 3/4] NIP-96: no transform (#1262) * no_transform * Update 96.md Co-authored-by: Santos <34815293+sant0s12@users.noreply.github.com> --------- Co-authored-by: Santos <34815293+sant0s12@users.noreply.github.com> --- 96.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/96.md b/96.md index e882870e..c8b3170b 100644 --- a/96.md +++ b/96.md @@ -101,6 +101,7 @@ List of form fields: - `alt`: **RECOMMENDED** strict description text for visibility-impaired users. - `media_type`: "avatar" or "banner". Informs the server if the file will be used as an avatar or banner. If absent, the server will interpret it as a normal upload, without special treatment. - `content_type`: mime type such as "image/jpeg". This is just a value the server can use to reject early if the mime type isn't supported. +- `no_transform`: "true" asks server not to transform the file and serve the uploaded file as is, may be rejected. Others custom form data fields may be used depending on specific `server` support. The `server` isn't required to store any metadata sent by `clients`. @@ -110,6 +111,8 @@ The hash is enough to uniquely identify a file, that's why it will be used on th The `server` MUST link the user's `pubkey` string as the owner of the file so to later allow them to delete the file. +`no_transform` can be used to replicate a file to multiple servers for redundancy, clients can use the [server list](#selecting-a-server) to find alternative servers which might contain the same file. When uploading a file and requesting `no_transform` clients should check that the hash matches in the response in order to detect if the file was modified. + ### Response codes - `200 OK`: File upload exists, but is successful (Existing hash) From 7bf5e327f7c0fef06173b10c3300767acd20d884 Mon Sep 17 00:00:00 2001 From: kieran Date: Wed, 29 May 2024 14:26:00 +0100 Subject: [PATCH 4/4] update list response --- 96.md | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/96.md b/96.md index c8b3170b..2f253516 100644 --- a/96.md +++ b/96.md @@ -286,26 +286,29 @@ Returns a list of files linked to the authenticated users pubkey. Example Response: ```js -[ - { - "id": "", - "nip94_event": { +{ + "count": 1, // server page size, eg. max(1, min(server_max_page_size, arg_count)) + "total": 1, // total number of files + "page": 0, // the current page number + "files": [ + { "tags": [ + ["ox": "719171db19525d9d08dd69cb716a18158a249b7b3b3ec4bbdec5698dca104b7b"], + ["x": "5d2899290e0e69bcd809949ee516a4a1597205390878f780c098707a7f18e3df"], ["size", "123456"], ["alt", "a meme that makes you laugh"], - ["expiration", "1715691139"] + ["expiration", "1715691139"], // ...other metadata ] - "content": "haha funny meme" // caption - } - }, - ... -] + "content": "haha funny meme", // caption + "created_at": 1715691130 // upload timestmap + }, + ... + ] +} ``` -`` is the **original hash**, ie. `ox` - -`nip94_event` is the same as in the upload result. +`files` contains an array of NIP-94 events ### Query args