Scanning Exchanges for Available Crypto Market Prices Data

  • 560 views
  • 5 minute read
Share
0 people shared the story
0
0
0
0
0
0
0
0
0
0
0

One of the most common problem encountered by all novice researches of the crypto-markets and (algo-)traders is knowing a list of all cryptocurrency pairs being actively traded at specific crypto exchange. This knowledge is a gateway to a vast research over correlations of crypto-assets, looking for arbitrage trading opportunities, targeting exchanges with most liquid assets or lowest spreads, etc.

In this post we will see how one can scan a selected crypto-exchange (Bitfinex) for available data, in particular the daily close prices. The broader context of this solution with some examples of its applications can be found in my upcoming book Cryptocurrencies with Python.

To begin, we will make use of the resources provided by CryptoCompare. Thanks to their API, it is easy to figure out how to employ Python in search for relevant data. Let’s import few libraries followed by reaching for an information pack covering the most relevant crypto-exchanges:

# Scanning Crypto Exchanges for Available Cryptocurrency Price Data
#
# (c) 2020 QuantAtRisk.com, by Pawel Lachowicz, PhD

import ccrypto as cc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import requests
import json
from bs4 import BeautifulSoup

url = "https://min-api.cryptocompare.com/data/v4/all/exchanges?topTier=1"
        
# fetch the raw data
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
dic = json.loads(soup.prettify())  # convert from json to dictionary
print(dic)

In return, at the first glance, we will receive the feedback of a rich structure:

{'Response': 'Success',
 'Message': '',
 'HasWarning': False,
 'Type': 100,
 'RateLimit': {},
 'Data': {'exchanges': {'ACX': {'isActive': False,
    'isTopTier': True,
    'pairs': {'PLA': {'tsyms': {'AUD': {'histo_minute_start_ts': 1575244800,
        'histo_minute_start': '2019-12-02',
        'histo_minute_end_ts': 1604102400,
        'histo_minute_end': '2020-10-31'},
       'USDT': {'histo_minute_start_ts': 1575244800,
        'histo_minute_start': '2019-12-02',
        'histo_minute_end_ts': 1604102400,
        'histo_minute_end': '2020-10-31'}}},
     'ETC': {'tsyms': {'AUD': {'histo_minute_start_ts': 1571356800,
        'histo_minute_start': '2019-10-18',
        'histo_minute_end_ts': 1604102400,
        'histo_minute_end': '2020-10-31'}}},
     'BTC': {'tsyms': {'USDT': {'histo_minute_start_ts': 1533859200,
        'histo_minute_start': '2018-08-10',
        'histo_minute_end_ts': 1604102400,
        'histo_minute_end': '2020-10-31'},
       'AUD': {'histo_minute_start_ts': 1571356800,
        'histo_minute_start': '2019-10-18',
        'histo_minute_end_ts': 1604102400,
        'histo_minute_end': '2020-10-31'}}},
     'XRP': {'tsyms': {'AUD': {'histo_minute_start_ts': 1571356800,
        'histo_minute_start': '2019-10-18',
        'histo_minute_end_ts': 1604102400,
        'histo_minute_end': '2020-10-31'}}},
...

Since we are interested in data available for all exchanges, we can limit ourselves just to:

d = dic['Data']['exchanges']
print(d)

i.e.

{'ACX': {'isActive': False,
  'isTopTier': True,
  'pairs': {'PLA': {'tsyms': {'AUD': {'histo_minute_start_ts': 1575244800,
      'histo_minute_start': '2019-12-02',
      'histo_minute_end_ts': 1604102400,
      'histo_minute_end': '2020-10-31'},
     'USDT': {'histo_minute_start_ts': 1575244800,
      'histo_minute_start': '2019-12-02',
      'histo_minute_end_ts': 1604102400,
      'histo_minute_end': '2020-10-31'}}},
   'ETC': {'tsyms': {'AUD': {'histo_minute_start_ts': 1571356800,
      'histo_minute_start': '2019-10-18',
      'histo_minute_end_ts': 1604102400,
      'histo_minute_end': '2020-10-31'}}},
   'BTC': {'tsyms': {'USDT': {'histo_minute_start_ts': 1533859200,
      'histo_minute_start': '2018-08-10',
      'histo_minute_end_ts': 1604102400,
      'histo_minute_end': '2020-10-31'},
     'AUD': {'histo_minute_start_ts': 1571356800,
      'histo_minute_start': '2019-10-18',
      'histo_minute_end_ts': 1604102400,
      'histo_minute_end': '2020-10-31'}}},
   'XRP': {'tsyms': {'AUD': {'histo_minute_start_ts': 1571356800,
      'histo_minute_start': '2019-10-18',
      'histo_minute_end_ts': 1604102400,
...

where

print(len(d.keys()))

68

will tell us how many Top Tier crypto-exchanges are currently out there (i.e. at the time of running the code). We can list them all:

i = 0
for e in d.keys():
    if d[e]['isActive']:
        i += 1
        print('%5gt%s' % (i, e))

    1	BTCAlpha
    2	BTCMarkets
    3	BTCTurk
    4	Bibox
    5	Binance
    6	Binanceje
    7	Bit2C
    8	BitBank
    9	BitBay
   10	BitMart
   11	BitTrex
   12	Bitfinex
   13	Bithumb
   14	Bitso
   15	Bitstamp
   16	Bleutrade
   17	Cexio
   18	CoinCorner
   19	CoinDeal
   20	CoinFalcon
   21	CoinJar
   22	Coinbase
   23	Coincheck
...
   65	huobijapan
   66	huobikorea
   67	itBit
   68	lmax

In next step, we need to select the exchange of our interest. Let’s assume we want to focus on Bitfinex exchnage and find out what cryptocurrencies are/were quoted vs USD, traded there, and CryptoCompare has the records of them:

coins = list()

for e in d.keys():
    if d[e]['isActive']:
        if e == 'Bitfinex':
            for p in d[e]['pairs']:
                x = list()
                for f in d[e]['pairs'].values():
                    x.extend(list(f['tsyms'].keys()))
                    
                if 'BTC' in set(x):
                    coins.append(p)
                    
coins = sorted(coins)
print(coins)

['ABYSS', 'ADA', 'AGI', 'AID', 'AION', 'ALGO', 'AMPL', 'ANT', 'AST', 'ATMI', 'ATOM', 'AUC', 
'AUTON', 'AVAX', 'AVT', 'BAL', 'BAND', 'BAT', 'BCH', 'BCI', 'BFT', 'BFX', 'BNN', 'BNT', 
'BOX', 'BSV', 'BT1', 'BT2', 'BTC', 'BTG', 'BTSE', 'BTT', 'CBT', 'CFI', 'CHZ', 'CLO', 'CND', 
'CNH', 'CNN', 'COMP', 'CRDTS', 'CSEN', 'CTXC', 'DADI', 'DAI', 'DAPP', 'DASH', 'DATA', 'DGB', 
'DGX', 'DOT', 'DOT_NO_SPLIT', 'DRGN', 'DTA', 'DTH', 'DTX', 'DUSK', 'EDO', 'EGLD', 'ELF', 
'ENJ', 'EOS', 'EOSDT', 'ESS', 'ETC', 'ETH', 'ETP', 'EURS', 'EURT', 'FET', 'FIL', 'FOAM', 
'FSN', 'FTT', 'FTXT', 'FUN', 'GENS', 'GNO', 'GNT', 'GOT', 'GT', 'GUSD', 'HOT', 'HYPERS', 
'IMPT', 'INT', 'IOST', 'IQ', 'JST', 'KAN', 'KNC', 'KSM', 'LEO', 'LINK', 'LOOM', 'LRC', 
'LTC', 'LYM', 'MANA', 'MDCL', 'MGO', 'MIOTA', 'MITH', 'MKR', 'MLN', 'MXAI', 'NCASH', 'NEC', 
'NEO', 'NIO', 'NUT', 'ODE', 'OKB', 'OMG', 'OMNI', 'ONL', 'ORS', 'PASS', 'PAX', 'PNK', 'PNT', 
'POA', 'POLY', 'PPAI', 'QASH', 'QTUM', 'RBTC', 'RCN', 'RDNN', 'REP', 'REQ', 'RIF', 'RINGX', 
'RLC', 'RRB', 'RRT', 'RTE', 'SAI', 'SAN', 'SEER', 'SNGLS', 'SNT', 'SNX', 'SPANK', 'STORJ', 
'SWM', 'TKN', 'TNB', 'TRIO', 'TRX', 'TUSD', 'UFR', 'ULTRA', 'UNI', 'USDC', 'USDK', 'USDT', 
'UTK', 'UTNP', 'VEE', 'VET', 'VLD', 'VSYS', 'WAX', 'WBTC', 'WLO', 'WPR', 'WTC', 'XAUT', 
'XCHF', 'XLM', 'XMR', 'XRA', 'XRP', 'XTZ', 'XVG', 'YEED', 'YFI', 'YOYOW', 'ZB', 'ZCN', 
'ZEC', 'ZIL', 'ZRX']

This is a lot!

The last step it to fetch the historical time-series of daily close prices for all abovementioned cryptocurrencies vs USD, and by doing so, check the actual data span:

data = pd.DataFrame(columns=['Cryptocurrency', 'From', 'To'])

k = 0
for coin in coins:
    try:
        df = cc.getCryptoSeries(coin, 'USD', freq='d', exch='Bitfinex')

        if df.size > 0:
            print("%7s%12s%12s" % (coin, df.index[0].strftime("%Y-%m-%d"), 
                                   df.index[-1].strftime("%Y-%m-%d")))
            data.loc[k] = (coin, df.index[0].strftime("%Y-%m-%d"), 
                           df.index[-1].strftime("%Y-%m-%d"))
            k += 1
    except:
        pass


data.From = pd.to_datetime(data.From)
data.To = pd.to_datetime(data.To)

Here, we construct an empty DataFrame where we want to store the results. In a loop, we go over all coins (alternatively speaking: cryptocurrencies) identified in the previous step. For each one, we use getCryptoSeries function from ccrypto library (available with the release of my book Cryptocurrencies with Python) to fetch the daily close price time-series for selected cryptocoin vs USD. Based on them, we check at what day the records begin and at what day they end.

The above loop, as of time of writing, returns:

  ABYSS  2018-09-01  2020-10-31
    ADA  2020-08-06  2020-10-30
    AGI  2018-05-09  2020-10-29
    AID  2018-01-30  2020-10-29
   AION  2018-05-09  2020-10-31
   ALGO  2019-07-01  2020-10-31
   AMPL  2019-06-28  2020-10-30
    ANT  2018-05-09  2020-10-31
    AST  2019-05-29  2020-10-27
...
  YOYOW  2017-12-01  2020-10-31
     ZB  2019-11-02  2020-10-30
    ZCN  2018-08-29  2020-10-31
    ZEC  2016-10-29  2020-10-29
    ZIL  2018-09-05  2020-10-23
    ZRX  2018-02-05  2020-10-31

For the grand finale, let us visualise data availability between 2015 and 2020:

# reverse the order of rows in DataFrame for plotting only
dataV = data.iloc[::-1]
dataV.index = np.arange(dataV.shape[0])

# plot
plt.figure(figsize=(12,40))
for i in np.arange(dataV.shape[0]):
    dr = pd.date_range(dataV.From[i], dataV.To[i])
    yr = np.zeros(len(dr)) + i
    plt.plot(dr, yr, 'r-', label=dataV.Cryptocurrency[i])

plt.yticks(np.arange(dataV.shape[0]), dataV.Cryptocurrency)
plt.grid()
plt.xticks(rotation=45)
plt.ylim([-1, dataV.shape[0]])


In this way, we obtain a quick overview of time span of all available cryptocurrencies quoted vs USD traded at Bitfinex crypto-exchange. In many instances, it’s extremely needed solution.

Although it does not check whether all daily price quotes are available for every cryptocurrency pair but, at least, it allows to understand the lifetime of specific cryptocurrency on the Bitfinex market.

Related Tags

Writes on quantitative finance, trading, algo-trading, crypto markets, risk management, and Python programming in finance since 2012. Sailor and proud father.