Query capabilities...

[Paulo Aboim Pinto (aboimpinto)]

Hello,
I would like to know the query capabilities that I can use.

From the example, I can only get an item from his key. Is there a way to get the sequential list of keys? How many items we have?

thanks
Paulo Aboim Pinto

[Badrish Chandramouli (badrishc)]

Currently supported:

• Get item given key (session.Read)
• Iterate all distinct key-value pairs in the DB (session.Iterate)
• Scan through all entries in record log, including duplicates (store.Log.Scan)
• Number of active entries in hash table (store.EntryCount) - this is approximate # distinct keys modulo hash collisions, for exact # use Iterate.

Work in progress:

• Secondary index framework for FASTER
• Secondary hash value index: query on secondary attributes (e.g., value.color == "blue")

Future work:

• Key range queries via secondary index

[Paulo Aboim Pinto (aboimpinto)]

Thanks, I will take a look at this ...

[cmadh]

Hey Badrish Chandramouli (@badrishc) what is the status and timeline for Faster Preveiw v2, the Secondary Indexing Framework and Hash Value Indices?

We would like to work with Faster but we will definitely need this functionality, I looked at the latest commits and branches but it doesn't look like the additional work regarding indices has started, is that correct or have I missed something?

Big thanks in advance, really like this project and your work, following it for a while.

[Badrish Chandramouli (badrishc)]

The FASTER v2 preview should be out soon, if not this week then by the first few weeks of Jan. The secondary indexing work is actually nearly complete, but it has not been converted into a public pull request just yet. It will not be a part of first v2 preview, though. We plan to create the indexing PR separately around January.

[cmadh]

Perfect, thank you for the quick response, can't wait to see and try the new indexing-features.

[cmadh]

Hey Badrish Chandramouli (@badrishc) , I see that there is a lot of progress in the Faster-repo currently but also that "Secondary Indexing (SI) Framework for FASTER" moved to v2.1. Is there a rough timeline for releases of v2 preview and also v2.1 and the SI Framework?

[Badrish Chandramouli (badrishc)]

v2 preview is already available, but we are stabilizing right now. plan to go GA in around a week. for 2.1, we do not have a timeline yet, but the SI framework should be in within 2 months from now.

[cmadh]

Hey Badrish Chandramouli (@badrishc) , i don't want to bug you but do you have new infos regarding the SI-framework? A rough timeline would help us with internal planning, even if it's delayed a few more months, but without any information internal planning is difficult...

[David Kamunyu (BrokenHelmet)]

Hi Badrish Chandramouli (@badrishc).

I just wanted to quickly inquire what the current status of FASTER is 🙏

[Badrish Chandramouli (badrishc)]

FASTER is quite stable at the moment, and we are working with various use cases internally and externally. The latest v2 of FASTER provides a very flexible IFunctions interface that will allow you to add in a secondary index yourself if you wanted to. Our in-built secondary index support is currently on hold pending other work items, we plan to get back to it within a month or two from now.

[David Kamunyu (BrokenHelmet)]

Alright, thanks for the update 👍

[Badrish Chandramouli (badrishc)]

cmadh - can you describe exactly what you are looking for in FASTER's secondary index support, and we can see how to help. cc Ted Hart (@TedHartMS)

[cmadh]

Thank you for checking back badrish and sry for the late response.

As a brief overview:
Our solution indexes historical and live data from several high-performance blockchains (yes, there are high-performance blockchains) which process a maximum of 10,000 transactions per second, each with several so-called actions (smart contract method invocations). There are several solutions based on different storage backends (Elasticsearch, Postgres, TiDB, etc.) but usually the big bottleneck is still the storage backend.

We work with several persistent FasterKV-Stores in parallel where individual data sets in different stores have a relation (blocks<-1---n->transactions<-1---n->actions) to each other. The required Storage of the Stores exceeds the available RAM by far, currently all stores together are between 15 and 20 TB in size depending on the blockchain beeing processed.

We work with non-blittable types and custom serializers/deserializers (we basically read binary data from the standard-output of the blockchain-node-process and process and store it). Overall, the solution works very well so far, it's extremly performant in both reads and writes and we have been able to solve some problems of similar solutions while improving various aspects, leading into higher efficiency and lower costs while still prototyping.

Secondary Indexing:
Currently we are mainly looking for a way to add secondary indices to our datasets in addition to the primary index which allow us to query datasets using these secondary indices.

Typically these secondary indices will be non-unique primitive values (long or ulong) and the number of records with the same secondary-key can be very high while more records with the same secondary-key can be added very frequently (there are smart contract actions called tens to hundrets of times per block (with a new block beeing created every 0.5 seconds)). The amount of secondary indices needed at the moment is between 1 and 4 per Store.

We could probably implement a simple SI mechanism ourselves by adding secondary index stores that contain all primary keys for each secondary key, but this would most likely lead to a greatly increased amount of updates/RMWs (indexing the "action_name" and/or "smart_contract_name" and/or the "sender_account" of hundrets of called actions per block would lead into continous RMWs) and we assume that it would significantly reduce the performance of the application and in addition we would have to implement our own form of pagination for the retrieval of datasets queried through the SI-Stores.

Currently, we are leaning towards using Lucene as a workaround solution while we wait for the Faster SI framework.

General
There are several reasons why we chose Faster as the main supported storage backend (we are also working on supporting other storage backends), but one of the main reasons was the planned FishStore integration.

Among other things, low latency binary streaming support of filtered data is planned for which PSFs, subset hashing and the functionality mentioned in the FishStore paper would be of great use. Also we would like to provide a less static way to define secondary indices and PSF-based scans where hosters of our Service can in the best case define PSFs on their own.

Side-Note: Our Solution is Open Source and can be found here: https://github.com/liquiidio/dfuse-light We are updating the public Repo every few weeks.

[cmadh]

Badrish Chandramouli (@badrishc) Ted Hart (@TedHartMS) I assume there are still no news regarding the SI-Framework? No Alpha, no Roadmap or so?