Spheron Infrastructure

Introduction

In this tutorial, we will learn how to use IPFS via Spheron Network in our task. While Spheron does support various other protocols, the IPFS protocol is mandatory for retrieving Content Identifiers (CIDs).

What is Spheron?

Spheron Network is a decentralized platform for automating and enhancing Dapps, focusing on rapid deployment, scalability, and personalized content delivery in a decentralized network, thereby overcoming the constraints of traditional centralized cloud services. Check out their website for more information.

Example Task

If you prefer learning by example, check out our X-Scraper Repo. It utilizes Spheron to store data in IPFS.

Environment Setup

YAML Configuration

Let's start from the beginning. How will your task-runners use Spheron on their end? Koii provides a one-click solution for that problem with our partnership with Spheron.

Open your config-task.yml file.
Under the requirementsTags section, you need to add a new entry for defining the SECRET_SPHERON_TOKEN variable.

requirementsTags:
  # Existing entries for CPU, RAM, STORAGE, etc. are above
  # Add the new TASK_VARIABLE entry below all existing entries
  - type: TASK_VARIABLE
    value: "SECRET_SPHERON_TOKEN"
    description: "used to store the data"
    retrievalInfo: "https://faucet-api.koii.network/api/get-spheron-storage-key;GET;SYSTEM_WALLET_PUBLIC_KEY"

By utilizing this variable, your task-runners will be able to obtain Spheron tokens within the Koii App with ease, by clicking the "get a key" prompt before running your app.

This way, you can use Spheron in your task without worrying about setting up the infrastructure by yourself.

Available retrievalInfo options are: URL;GET|POST;SYSTEM_WALLET_PUBLIC_KEY|STAKING_WALLET_PUBLIC_KEY

Required Package

With that out of the way, we can now focus on the code implementation.

First, we need the @spheron/storage package.

npm install @spheron/storage

Task Implementation

Imagine that we have a task that scrapes data from a public website and stores it in a JSON object. Our node has already followed the required steps for web scrapping, and now we want to store this JSON file in IPFS via Spheron, and utilize the CID as a part of our submission.

Important Note

Currently, Spheron does not support direct JSON uploads. Only files are supported. This means that we will need to convert our JSON data into a file before uploading it to Spheron.

In Node.js environments, the absence of the File module (which is a browser-specific API) means that we need to use the fs module to create a file from our JSON data.

Once written, the file's path is then provided to the client.upload() method in Spheron, along with the specified protocol and file name. This method returns the unique CID for the uploaded file.

Here is an example function that will create a file from a JSON object.

const {
  namespaceWrapper,
} = require("(Namespace Wrapper Path found within the Koii Template)");
const { writeFileSync } = require("fs");

async function makeFileFromObjectWithName(obj) {
  try {
    const dataString = JSON.stringify(obj);
    const path = await namespaceWrapper.getBasePath();
    const filePath = path + "/data.json";
    writeFileSync(filePath, dataString);
    return filePath;
  } catch (error) {
    console.log("error", error);
  }
}

And here is another snippet that will upload the file to Spheron.

function makeStorageClient() {
  const { SpheronClient, ProtocolEnum } = require("@spheron/storage");

  return new SpheronClient({
    token: process.env.Spheron_Storage,
  });
}

async function storeFiles(filePath) {
  const client = makeStorageClient();

  let currentlyUploaded = 0;

  const { cid } = await client.upload(filePath, {
    protocol: ProtocolEnum.IPFS,
    name: "taskData",
    onUploadInitiated: (uploadId) => {
      console.log(`Upload with id ${uploadId} started...`);
    },
    onChunkUploaded: (uploadedSize, totalSize) => {
      currentlyUploaded += uploadedSize;
      console.log(`Uploaded ${currentlyUploaded} of ${totalSize} Bytes.`);
    },
  });

  console.log(`CID: ${cid}`);
  return cid;
}

danger

Please do not change name: "taskData", as it is a mandatory field for Spheron. If you change it, your task may fail.

Now you have access to the CID of your file, which is the key to obtaining your JSON file from anywhere else. You can use this CID as a part of your submission, which will allow you to retrieve the data back.

Data Retrieval from IPFS

In order to retrieve the data, we need to know the CID of the file. In our case, we will use the CID as a part of our submission. This way, we can retrieve the data from the CID that is stored in the blockchain.

Here is an example code snippet that retrieves the data from a CID.

const axios = require('axios');

const getJSONFromCID = async (cid, fileName, maxRetries = 3, retryDelay = 3000) => {
  let url = `https://${cid}.ipfs.dweb.link/${fileName}`;
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const response = await axios.get(url);
      if (response.status === 200) {
        return response.data;
      } else {
        console.log(`Attempt ${attempt}: Received status ${response.status}`);
      }
    } catch (error) {
      console.log(`Attempt ${attempt} failed: ${error.message}`);
      if (attempt < maxRetries) {
        console.log(`Waiting for ${retryDelay / 1000} seconds before retrying...`);
        await sleep(retryDelay);
      } else {
        return false;
      }
    }
  }

Introduction​

Environment Setup​

YAML Configuration​

Required Package​

Task Implementation​

Data Retrieval from IPFS​