Importing historic data from Kraken

lightyear15

Hello,
I am trying to implement the CandleStackExchange driver for Kraken exchange.
Unfortunately, unlike binance, kraken provides only the past 720 candles at a given time interval.
On the other hand kraken provides the entire history of all the trades sent to their market.
Sample of a trade data is like :

[
"9157.00000", # price
"0.00000047", # amount
1590242415.6565, # tstamp
"s", # s for sell / b for buy
"m", # m for market / l for limit
"" # ignored
],

Now, does anybody have any suggestion on how to convert these data into candles?
I am not asking for code samples or detailed algorithm, just a rough sketch of procedure.
Do I need to rebuild the entire book at each time instant and then from there build list of transactions, ecc...?

Thanks

P.S. for any reference
https://www.kraken.com/features/api
https://support.kraken.com/hc/en-us/articles/218198197-How-to-retrieve-historical-time-and-sales-trading-history-using-the-REST-API-Trades-endpoint-

Saleh

Hey

I took at quick look at their docs. I don't think you'd need trades to create candles from. This seems to be the endpoint for what you need:
https://www.kraken.com/features/api#get-ohlc-data

kraken provides only the past 720

Don't worry about that. In fact the current Coinbase provides only 300 per request. Look at the code for Coinbase driver and try to keep the same structure and you'd be fine.

lightyear15

alright,
thanks.
Hope I did a decent job.
Can't wait to add support for live mode...
https://github.com/jesse-ai/jesse/pull/23

lightyear15

Hey @Saleh ,
I have a new version of the Kraken driver, capable of importing historic data, it uses trade endpoint and reconstruct candles using pandas.DataFrame.group_by functions.
It's tested, I have imported data since October 2019 for many currencies.
Importing typically takes very long since the endpoint provides only 1000 trades per request and for cryptos like BTC, 1000 trades might cover only a few minutes of history.
That is why I was thinking about implementing an extra confirmation message for drivers, default to empty str, except for the kraken one where I would write something like "This might take very long time, blahblah...".
Would you like to have a look at it?
(Just asking here before opening any PR and create pointless noise)

Saleh

lightyear15 Hi mate.

Kraken seems too much of a trouble. I would like to stay away from it for now. I might instead add an interface or some kind of plugin system for anyone to add their own functionality. That seems easier to me than adding a code that I don't fully understand (maintaining it would be hard for me)

lightyear15

Saleh
Alright,
let me know if I can help with something 👍️

Saleh

lightyear15 Will do. I appreciate it 🙏

Gabri

Hey @Saleh, I'd like to try to add FTX exchange support to Jesse. I've seen the content of drivers folder ..and it came to me a question: what about the websocket implementation?
Are those REST drivers therefore used only for backtests?
What about live/paper trade mode and probably the websocket implementation? There will be need a specific implementation for every exchange too, right?
Thanks

markus

Gabri As far as I'm up to date you are right. Those drivers are only for the import/backtest. That's why they are within the import_candles_mode subfolder. Live/Paper mode will have its own drivers and I think its a good approach to separate them. For example, you might have drivers for candle import that use services that in contrast to exchanges just offer candles data and nothing more. Does that make sense?

lightyear15

Hey @Gabri,
have a look at my PR
https://github.com/jesse-ai/jesse/pull/53
it's not mergeable yet, it misses unit tests, yet it might be a good starting point for you to implement your candle importer.

As Markus said, candle importer just import candles, they are not meant to support live/paper trading, for that, there will be a new driver interface... I suppose

Saleh

lightyear15 I really like to dig into it as soon as I can. The idea of a plugin based design to allow everyone to create heir own driver is awesome. It's just I've been very busy lately.

Gabri

lightyear15 Thank you, I'll start from there! 😀
Ok, in a second time there will the chance to implement the websocket version in a way or in another (so far I'll study just the REST API documentation)