Dealing with ambiguity in token symbols

Introduction

This post tries to explain why Rotki’s approach to dealing with ambiguity in crypto assets is needed and why other approaches are error prone and most probably lead to broken results.

We will see what Rotki does in detail and what challenges exist when interfacing with multiple exchanges and other external services.

Understanding The Problem

The best way to understand the problem is by example. Take the cryptoasset with the symbol KEY. Most of our competitors would simply see you have a KEY balance and query for its price at any given moment from a website such as cryptocompare.com.

The problem is that as of the writing of this post three different tokens exist that use that symbol.

To make matters worse, price aggregator websites such as cryptocompare and coinpaprika have different representations.

For example:

  • SelfKey is known as KEY in cryptocompare
  • Bihu Key is known as BIHU cryptocompare
  • KeyCoin is known as KEYC in cryptocompare

  • SelfKey is known as key-selfkey in coinpaprika
  • BihuKey is known as key-key in coinpaprika
  • KeyCoin is not known in coinpaprika

Add different representations for each symbol in different exchanges in the mix and then the whole situations gets even more entangled as we can see with ETHOS and BQX below.

Do you believe that competitor webapps take all these different situations into account? Want a bet? Let’s go and check their implementation … oh wait … we can’t. Not only do they need you to upload all your data to them but their sourcecode is not open and thus you can’t audit their calculations.

Your SelfKey could be priced as Keycoin or viceversa, or worse bail out since it can’t find a price and not consider it in any profit/loss calculations.

Solving the Problem

The only way to handle this problem is by maintaing a database of all assets that are supported. Rotki does this by maintaining the all_assets.json file.

What are the rules for an asset to get listed? Rather simple actually:

  1. If a user asks for an asset it will be added as long as a price for it exists in a supported price aggregator website (for the moment either in coinpaprika or cryptocompare).
  2. If it’s listed in any of the supported exchanges.
  3. In both of the above cases historical prices need to also be discoverable in a price aggregator website. The lack of price discovery is for example why the STL token PR is blocked as of the time of writing this post.

For assets with identical symbols like KEY we end up with the following in assets.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
"KEY": {
    "ethereum_address": "0x4CC19356f2D37338b9802aa8E8fc58B0373296E7",
    "ethereum_token_decimals": 18,
    "name": "Selfkey",
    "started": 1508803200,
    "symbol": "KEY",
    "type": "ethereum token"
},
"KEY-2": {
    "ethereum_address": "0x4Cd988AfBad37289BAAf53C13e98E2BD46aAEa8c",
    "ethereum_token_decimals": 18,
    "name": "Bihu KEY",
    "started": 1507822985,
    "symbol": "KEY",
    "type": "ethereum token"
},
"KEY-3": {
    "active": false,
    "ended": 1452038400,
    "name": "KeyCoin",
    "started": 1405382400,
    "symbol": "KEY",
    "type": "own chain"
},

Asset Entry

All entries are comprised of a unique asset identifier, which is the key in the above JSON object. The identifier is always the symbol of the asset, and if an asset with the same symbol already exists then the identifier get a number prefix as seen above.

The rest of the attributes will be explored below. The attributes of an asset may change in future iterations of Rotki but as of v1.0.3 it’s the following:

name

This is a required attribute. It’s the name by which the asset is commonly known.

symbol

This is a required attribute. It’s the symbol the asset has. This is not guaranteed to be unique across all the supported assets as is also made clear by the KEY token example.

type

This is a required attribute. It’s the type of asset this is. Determines if it’s a blockchain asset and if yes on which chain it is. Valid values as of writing this post can be seen here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
asset_type_mapping = {
    'fiat': AssetType.FIAT,
    'own chain': AssetType.OWN_CHAIN,
    'ethereum token and own chain': AssetType.OWN_CHAIN,
    'ethereum token and more': AssetType.ETH_TOKEN_AND_MORE,
    'ethereum token': AssetType.ETH_TOKEN,
    'omni token': AssetType.OMNI_TOKEN,
    'neo token': AssetType.NEO_TOKEN,
    'counterparty token': AssetType.XCP_TOKEN,
    'bitshares token': AssetType.BTS_TOKEN,
    'ardor token': AssetType.ARDOR_TOKEN,
    'nxt token': AssetType.NXT_TOKEN,
    'Ubiq token': AssetType.UBIQ_TOKEN,
    'Nubits token': AssetType.NUBITS_TOKEN,
    'Burst token': AssetType.BURST_TOKEN,
    'waves token': AssetType.WAVES_TOKEN,
    'qtum token': AssetType.QTUM_TOKEN,
    'stellar token': AssetType.STELLAR_TOKEN,
    'tron token': AssetType.TRON_TOKEN,
    'ontology token': AssetType.ONTOLOGY_TOKEN,
    'exchange specific': AssetType.EXCHANGE_SPECIFIC,
    'vechain token': AssetType.VECHAIN_TOKEN,
    'binance token': AssetType.BINANCE_TOKEN,
}

active

The active attribute is optional. If missing, a true value is implied. It signifies if the asset is actively traded in any exchange and has a price.

ended

This is an optional attribute but is required if an asset is not active. If an asset is not active this attribute signifies the timestamp at which all trading (and thus price) ceased for the asset.

ethereum_address

If the type of the asset is ethereum_token or related then it should also contain this entry. This entry contains the EIP55 encoded address of the token’s contract address in the main ethereum chain.

ethereum_token_decimals

Just like the previous entry if the asset is an ethereum token we also need to know its decimals in order to know how to display it to the user.

forked

This is an optional attribute. If the asset is a fork of another asset then the originating asset before the fork should be shown here.

For example BCH has the BTC forked attribute since it’s a fork off Bitcoin.

swapped_for

This is an optional attribute. If the asset ceased to exist but was swapped for another asset then this attribute points to the new asset.

For example SCJX was a counterparty token which got swapped for the STORJ ethereum token.

Conversions

With our list of assets at hand we need to be able to interact with other websites such as price aggregators or exchanges. The idea is that we maintain the Rotki database of assets which is our local “truth”. And when serializing our assets to communicate with another website or deserializing the assets we read from a website we need converters.

This is where the assets/converters.py module comes in.

For all incoming assets we convert to the Rotki format when necessary:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
def asset_from_kraken(kraken_name: str) -> Asset:
    if not isinstance(kraken_name, str):
        raise DeserializationError(f'Got non-string type {type(kraken_name)} for kraken asset')
    name = KRAKEN_TO_WORLD.get(kraken_name, kraken_name)
    return Asset(name)


def asset_from_cryptocompare(cc_name: str) -> Asset:
    return Asset(CRYPTOCOMPARE_TO_WORLD[cc_name])


def asset_from_poloniex(poloniex_name: str) -> Asset:
    if not isinstance(poloniex_name, str):
        raise DeserializationError(f'Got non-string type {type(poloniex_name)} for poloniex asset')

    if poloniex_name in UNSUPPORTED_POLONIEX_ASSETS:
        raise UnsupportedAsset(poloniex_name)

    our_name = POLONIEX_TO_WORLD.get(poloniex_name, poloniex_name)
    return Asset(our_name)


def asset_from_bittrex(bittrex_name: str) -> Asset:
    if not isinstance(bittrex_name, str):
        raise DeserializationError(f'Got non-string type {type(bittrex_name)} for bittrex asset')

    if bittrex_name in UNSUPPORTED_BITTREX_ASSETS:
        raise UnsupportedAsset(bittrex_name)

    name = BITTREX_TO_WORLD.get(bittrex_name, bittrex_name)
    return Asset(name)


def asset_from_binance(binance_name: str) -> Asset:
    if not isinstance(binance_name, str):
        raise DeserializationError(f'Got non-string type {type(binance_name)} for binance asset')

    if binance_name in UNSUPPORTED_BINANCE_ASSETS:
        raise UnsupportedAsset(binance_name)

    if binance_name in RENAMED_BINANCE_ASSETS:
        return Asset(RENAMED_BINANCE_ASSETS[binance_name])

    name = BINANCE_TO_WORLD.get(binance_name, binance_name)
    return Asset(name)

and the asset.py module itself has code to export from the Rotki format to all websites that need conversions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def to_kraken(self) -> str:
    return WORLD_TO_KRAKEN[self.identifier]

def to_bittrex(self) -> str:
    return WORLD_TO_BITTREX.get(self.identifier, self.identifier)

def to_binance(self) -> str:
    return WORLD_TO_BINANCE.get(self.identifier, self.identifier)

def to_cryptocompare(self) -> str:
    cryptocompare_str = WORLD_TO_CRYPTOCOMPARE.get(self.identifier, self.identifier)
    # There is an asset which should not be queried in cryptocompare
    if cryptocompare_str is None:
        if self.identifier == 'MRS':
            raise UnsupportedAsset(
                'Marginless is not in cryptocompare. Asking for MRS '
                'will return MARScoin',
            )
        else:
            raise RuntimeError(
                f'Got {self.identifier} as a cryptocompare query but it is '
                f'documented as returning None and is not handled',
            )

    return cryptocompare_str

Whenever a new asset is added and a conversion needs to be included then it is appropriately plugged into any of the above modules.

How to keep this all up to date?

There are two ways to keep all these mappings and the asset database up to date:

  1. Automated CI testing. We have tests for every exchange (for example here is binance) which will warn us when an exchange adds or changes something.

  2. Our users will be warned when they are trying to interact with something that is not supported or when something breaks and as such are incentivized to open issues at the Rotki repo.

Why not use other asset databases?

The reason is simple. That would be adding yet another conversion for us to maintain. For a project like Rotki, where certainty for what each symbol means needs to exist the only way to go is to have our own database as most of the currently known token databases out there are either ethereum specific or incomplete.

The only thing that could work is a standardized database of all crypto assets maintained by multiple different entities.

ITSA is trying to do something similar, but the entire dataset seems to be for “members only” and the way they operate seems to be non-transparent. We believe that any such standardization effort should be open-source and use collaborative tools such as Github.

It would be really neat to see people collaborate on Github in the Rotki database of assets and conversions. If that happens, perhaps it can become a standard that other projects can also use.

Conclusions

Working with multiple tokens across different websites is a hard problem. The Rotki approach took a lot of work to build but now that it’s there maintaining it is not that hard. Our database is open for everyone to use and contribute. If you want to edit information on a token or add a new one simply open a pull request.

Finally remember that Rotki is opensource and self-funded software. If you appreciate what we are doing please consider purchasing a premium subscription in order to help us keep developing and also enjoy premium only features such as analytics and priority support and feature requests.

Rotkehlchen's Values and Vision

Introduction

In this post we are going to see what Rotkehlchen is, what is it trying to solve, what is the value proposition of the application compared to competitors, the values we stand for, our vision for the future and more.

Our values

The casual reader may think: “Okay so what the heck am I here for? I see a hard to pronounce German name and a cute bird picture.”

Read on to see what is the problem we are trying to solve, how we do that and what our value proposition is.

What is the Problem?

Does the following situation sound familiar?

You have bought a few different cryptocurrencies, traded a bit, even bought something with your cryptocurrencies and now you are at a loss with what you have and where. In which account was Token X, did you deposit Token Y to Exchange Z and so on.

Also if you ever wanted an overview of the performance of your portfolio you have no idea how to even begin to calculate your profit/loss. What about taxes? How can you take into account all the different rules your jurisdiction imposes?

How does Rotkehlchen help?

This is where Rotkehlchen comes in.

It allows tracking of all your crypto assets, no matter if they are located on a blockchain account or on an exchange. It gives you an overview of all assets you own as well as provides you with the ability to analyze your transaction history in exchanges thus enabling calculation of profit/loss over a time period.

With a premium subscription it also provides very nice visualizations of the above in easy to read graphs such as the one below showcasing a test user’s total USD value in BTC and their total BTC amount graphed over time.

Our test user sold when BTC price was high to rebalance his portfolio and bought again when it was low according to the graph.

You can also customize the profit/loss calculation to adjust it to the requirements of your jurisdiction and get a detailed report which can be exported in CSV and handed over to your accountant.

Rotkehlchen’s value proposition

Okay so what? There are many other applications which offer similar services. cointracking.info cryptotax.de to name just a few.

It’s true. We are surrounded by coin tracking applications. We see a new one pop up every day. But they all share the same common flaw. They are centralized online applications onto which you have to relinquish all your data.

Arguably there is few data more important than the entire history of your financial transactions. And to simply hand it over to a centralized closed-source service is unthinkable! Especially for people who are prominent defendants of privacy and decentralization as most crypto users are.

Using any of the above applications is dangerous to you and to your financial privacy. Avoid them like the plague!

Here in Rotkehlchen we won’t have any of that and this is the strongest point in our value proposition:

  • Rotkehlchen is a local application available in all major Operating systems. You don’t need to send your data anywhere and worry about their eventual misusage. They are all local to your system.

  • Rotkehlchen is opensource. You don’t need to wonder what it does with your data, how it calculates what it does or even believe us at all. Just go to the source and verify that what we are saying here is true.

  • Rotkehlchen keeps all data encrypted locally. Your data is yours and encrypted locally with a password of your choice when you make your account.

  • Rotkehlchen is transparent and auditable. Since everything is open you will never have to question why something has been calculated as it is and neither will your accountant or your tax authority or whomever you choose to share your data with.

Our Vision

Our vision for tracking, accounting and analytics of finance is one of transparency, openness and empowering of the individual. The development of Rotkehlchen and our roadmap reflects those values. We want to enable you to take ownership of your financial data.

Planned Features

This is a rough outline of some of the features we have planned for Rotkehlchen both in the near but also long-term future.

Tighter integration with DeFi

A very hot topic in Ethereum these days is Decentralized Finance (DeFi). From lending protocols, decentralized exchanges, derivatives there is a lot of ways to get involved.

It is our goal to slowly but steadily integrate Rotkehlchen with as much of the DeFi ecosystem as possible so that you can track, analyze and interact with it through the privacy of Rotkehlchen’s local application.

Frontend rewrite in Vue.js

As seen here the frontend of the application will be rewritten in Vue.js. This will allow for prettier and more usable frontend and also better maintanability of the frontend code.

Track other financial assets

Rotkehlchen started as a crypto asset tracking and analytics application. But some of our users have also asked for the ability to track other more traditional assets such as stocks, real estate … even so far down as tracking every day expenses.

We would like to cater to most people’s needs and as such will slowly roll out support for extra assets as long as time and resources allow.

Android companion application

The need to check a portfolio on the-go or to add and track expenses arises pretty often. At the moment as a user you would have to wait until you get to your computer to open the Rotkehlchen application.

In order to solve this problem we would like to, in the long term, offer an android companion application that will allow tracking of every day expenses, monitoring of all your crypto assets and syncing with the main application.

Integrating with more Exchanges

There are so many exchanges out there that it’s hard to integrate with all of them. Each integration requires quite a bit of coding time in order to make sure that the API is integrated properly and that all assets offered by the exchange are supported.

We want to support as many exchanges as possible and so far these are the exchange support requests we have gotten.

We will prioritize on what most users need, judged by the number of thumbs up in each issue. If you would like to see Rotkehlchen support an exchange please put a thumbs up on the respective issue or create a new one if no issue exists for the exchange you would like to see supported.

Multiple cryptocurrency price sources.

At the moment we are solely relying on cryptocompare for cryptocurrency historical price data. They are by far the most advanced data provider out there but fallbacks are needed.

As can be seen by this issue we will integrate other data providers such as coin paprika and no longer rely on a single soure for price data.

What features do you need?

If there is something else that you would like to see added to Rotkehlchen please visit our issue tracker and open a feature request.

We want to build an application that you will find useful. Help us make Rotkehlchen an application you would love to use on a daily basis.

Supporting us

The vision of data ownership and transparency in accounting is something we strongly believe in. That is the reason we are an opensource local application, so that we can enable you to take ownership of your data and that you can transparently see what is being calculated and how.

Rotkehlchen Premium

Funding an opensource project is not an easy task. Our approach to funding and to at the same time maintaining our values is to offer a premium subscription on top of the normal Rotkehlchen experience.

With a premium subscription you get statistics, graphs and analytics, data sync between devices and more. For more information and to also purchase a premium subscription go here.

By purchasing a premium Rotkehlchen subscription not only do you get to enjoy extra features but you are funding opensource without the use of any middlemen! We accept both fiat and crypto currencies.

Please help us develop Rotkehlchen and protect your financial privacy.

Donations

If for some reason you don’t want to purchase a premium subscription but would still like to support us we accept cryptocurrency donations in ETH and BTC at the addresses seen here.

Tipping on Brave Browser

If you are using brave you can tip us with BAT both at our website and our twitter. For more information on brave tipping check the documentation.

Conclusion

Do not fall for all those centralized cointracking services that devour of your financial data and also have a rather steep price tag. Take ownership of your financial data today by using Rotkehlchen.