NFT Parser Specs

Details on how Pera's NFT parser scans for new NFTs.

(1) We have a continuously running task to fetch NFTs. This task iterates through all of the assets in our database endlessly. If we encounter an asset that has a URL field that:

1. Has an IPFS ID in it (an IPFS Gateway URL in this format: mysite.com/ipfs/<ipfs_id> or an IPFS URL in this format: ipfs://<ipfs_id>).

2. Is a URL that:

   1. Has http or https scheme

   2. Is missing a scheme, but becomes a valid URL after prepending https scheme

   3. Doesn't have a port defined

   4. Doesn't have a reserved IP address (for a list of reserved IP addresses, please see here)

then we asynchronously try to fetch the file from the asset URL (we are using a dedicated IPFS gateway for assets with IPFS URLs) if it is not fetched and saved before. Please note that files that have a size larger than 100 MB are ignored. If we encounter an error (for example the URL might not be working), we retry this asynchronous task 3 times. If it is still unsuccessful, our endlessly running task will retry to fetch this NFT in the next iteration.

(2) If we can successfully fetch the file from the URL, then using some third party packages (e.g. libmagic) we try to guess its mime type from its content. If the file is a JSON object (please note that a valid JSON can also be an array, a string or a number but we only accept objects) then we consider that asset as an ARC3 NFT, and the JSON as its metadata. If the file is an image or a video, we consider that asset as an ARC69 NFT, and the file as its media. Then we fire different asynchronous tasks for ARC3 and ARC69 NFTs. Please note that we don't repeatedly try to fetch the successfully fetched files.

(3a) For ARC3 NFTs, we try to fetch media of the NFT asynchronously using the metadata we have retrieved in the previous task. For the image of the NFT we try to get image field of the metadata, which should be the URL of the image, and try to fetch the file. If we can successfully fetch the file and validate that it is an image then we save it. For the video of the NFT we try to get animation_url field of the metadata, and try to fetch the file. If we can successfully fetch the file and validate that it is a video then we save it. An ARC3 NFT can both have an image and a video media at the same time. For both image and video URLs, unlike above we accept every URL in this case. Please note that we don't repeatedly try to fetch the image or the video if they are successfully fetched once. We only support videos and images currently.

(3b) For ARC69 NFTs, we try to fetch the metadata of the NFT asynchronously from the last acfg transaction with a note-prefix equal to "{" (to filter JSON objects). In each iteration we repeat this task since metadata of the ARC69 NFTs are mutable.

(4) Apart from our endlessly running task, if our block follower encounters an asset creation transaction, it triggers all of the steps above. If our block follower encounters an asset config transaction, it runs step 3b again if the asset has a collectible.

(5) We follow the ARC69 standards for traits: https://github.com/algorandfoundation/ARCs/blob/main/ARCs/arc-0069.md