nips/XX.md

NIP-XX: Decentralized file archival and retrieval

draft optional

This NIP defines how immutable files can be located and referenced in a way that is decentralized, flexible, extensible, and easy to implement.

It specifies:

1. A content-addressable URI scheme that can used in notes and messages to reference files and hints on where to find them. Clients can use this URI to retrieve the file from any server or system that supports it in a tamper-proof way.
2. A standard HTTP endpoint interface that can be optionally used by providers to serve these files to clients.

Motivation

Nostr currently relies on uploading and retrieving media through centralized servers. This NIP aims to provide a more decentralized approach without adding too much complexity, as well as to open doors to new applications and tools that want to deal with file archival and retrieval.

Here are the main issues with the current approach and other alternatives:

• Centralized media uploaders: Generally works, but presents the following issues:
  • Adds centralization to Nostr.
  • There is no way to verify the file was not tampered with or modified.
• IPFS: IPFS is decentralized, but:
  • it is not easy to integrate with. (The protocol is complex)
  • does not integrate well with other Nostr NIPs
  • it is not flexible enough to support other use cases
  • brings many other issues, which are well illustrated in this article

Use cases

• More decentralized Nostr media uploader services
• File integrity and tamper detection for clients.
• A gallery of photos that are stored on a local file system, and replicated across multiple personal servers
• A decentralized npm-like registry for software packages or git repositories
• Monetization of file access (e.g. Pay to download a book)
• Private caching servers for files (e.g. Save bandwidth and increase privacy by caching files to a personal server)

Specification

The following sections describe some primitive components of the spec. Clients and providers are free to implement one or many of these components or to integrate them with other specs or systems.

1. nblob bech32 entities

nblob are bech32 entities that identifies immutable file contents or any binary blob.

Obtaining the nblob of a file is done by:

1. hashing the file with SHA256
2. Encoding the hash as a bech32 string with the nblob prefix

Example implementation:

import hashlib
import bech32
import sys

def nblob(file):
    hash = hashlib.sha256(file).digest()
    return bech32.encode('nblob', 0, hash)
    
if __name__ == '__main__':
    file_path = sys.argv[1]
    with open(file_path, 'rb') as f:
        file = f.read()
        print(nblob(file))

Example nblob identifier of this script file: nblob1q9maw3n56tnvgqy2xaqzwgvjys5mptvh6hpffhwrpduc0r89pmr0q5k9p4t

2. nostr+file URI

nostr+file is a URI scheme that allows for referencing nblob entities.

The format of the URI is:

nostr+file:<nblob>?<query_parameters>

Where the optional URL-encoded query parameters can be:

• filename: the name of the file with its extension
• mime: the MIME type of the file following the IANA media types standard
• gateways: a comma-separated list of gateways that can be used to retrieve the file. It provides a hint (or recommendation) to the client on where to look for the file. The client MAY ignore this hint.

If both filename and mime are provided, the client SHOULD prefer the mime parameter over the file extension to determine the MIME type of the file.

Example nostr+file URI: nostr+file:nblob1q9maw3n56tnvgqy2xaqzwgvjys5mptvh6hpffhwrpduc0r89pmr0q5k9p4t?filename=example.py&gateways=https%3A%2F%2Fgateway.example.com%2Chttps%3A%2F%2Fgateway-example-2.com

These URIs can be pasted into Nostr notes, messages, articles or whatever other content that supports URIs.

Clients MAY use any system or method to retrieve the file from the nblob identifier. The following section describes a standard HTTP endpoint interface that can be optionally used by clients to retrieve the file.

3. HTTP Gateways

HTTP Gateways are servers that can be used by clients to retrieve files.

Providers that are interested in serving files over HTTP under this NIP MUST implement the following endpoint:

GET /.well-known/nostr/nipXX/<nblob_address>

Clients can use this endpoint to retrieve the file contents of a nblob address.

Clients MAY use the following HTTP request headers:

• Content-Type: To specify the MIME type (optional)
• Authorization: Optional NIP-98 authentication or payment authentication (e.g. Using L402)

If the file could be found on the server and the user is allowed to access it, the server MUST return the file contents on the response body.

HTTP response codes:

• 200: Found file and returned it
• 404: File not found
• 401: Unauthorized. Requires NIP-98 authentication
• 403: NIP-98 passed, but the Nostr user is not allowed to access this resource.
• 402: File access requires payment (using L402, for example)

HTTP response headers:

• Content-Type: Specifies the MIME type (Gateway SHOULD respect the MIME type requested by the client, but MAY show a different mime type if it is known by the server to be more accurate). Clients can use or ignore this header.

Once the file is retrieved, the client MAY verify the file contents by hashing it and comparing it to the nblob address.

Example flow

Here is an example flow illustrating all components of this NIP working together:

1. Alice calculates the nblob identifier for a picture she wants to share. She then uploads the same file to example.com and example2.com which are HTTP gateways she trusts.
2. Alice writes a NIP-01 text note and includes the URI: nostr+file:nblob1q9maw3n56tnvgqy2xaqzwgvjys5mptvh6hpffhwrpduc0r89pmr0q5k9p4t?filename=picture.jpg&gateways=https%3A%2F%2Fexample.com%2Chttps%3A%2F%2Fexample-2.com&mime=image%2Fjpeg in the note.
3. Bob reads the text note. Bob's client identifies and parses the nostr+file URI.
4. Bob's client then tries to send a GET request to the private gateway in their private home server at http://private.local/.well-known/nostr/nipXX/nblob1q9maw3n56tnvgqy2xaqzwgvjys5mptvh6hpffhwrpduc0r89pmr0q5k9p4t
5. Unfortunately, the file is not found in the private gateway. Bob's client then tries the recommended gateway by sending the GET request to https://example.com/.well-known/nostr/nipXX/nblob1q9maw3n56tnvgqy2xaqzwgvjys5mptvh6hpffhwrpduc0r89pmr0q5k9p4t, with Content-Type: image/jpeg (Inferred from the filename).
6. The server responds with HTTP 404 because the file is not found. (Let's assume the file was deleted from the server)
7. Bob's client then sends the same request to example2.com.
8. The server example2.com responds with HTTP 200 and returns the file contents with Content-Type: image/jpeg (in this case it also knows the MIME type of the file). Bob's client verifies the file contents by hashing it and comparing it to the nblob identifier.
9. Bob's client then displays the picture to Bob.

Items intentionally left out of this NIP

There are many ways this spec can be extended, but they are probably better tracked in separate NIPs. Here are some examples of things that are left out of this NIP on purpose:

• File upload endpoint: This NIP does not specify how files are uploaded to the network. There could be an extension to the gateway interface to allow for file uploads, but that is out of scope for this NIP.
• Directories: This NIP does not specify how to organize files into directories. This is left to other NIPs or to the implementer.
• Proof of storage: This NIP does not specify a provider can prove that they are storing the file without sending the file itself. This is left to other NIPs or to the implementer.
• Direct acyclic graphs of files: This NIP does not specify how to handle a DAG of files. This is left to other NIPs or to the implementer.
• Other discovery methods of gateways containing a file: Outside the hints provided in the gateways query parameter, this NIP does not specify other ways to discover gateways containing a file. This is left to other NIPs or to particular use cases.
• Serving files over other protocols: This NIP only specifies serving files over HTTP for simplicity. Other protocols (such as Websockets, SFTP, etc.) are left to other NIPs or to the implementer.