Version 0.37.0 is released, spot trading in backtests, trading other stablecoins in live mode, and more. BREAKING changes included, make sure to read

Today I released version 0.37.0 of the framework and the live plugin. This release brings part 1 of the long-awaited support for spot trading.

Part 1 brings support for spot trading in backtesting, and part 2 makes it possible in live trading.

It is a bold release as it includes new features and most importantly breaking changes. So reading the changelog of this release and updating instructions is a must.

What's new

[BREAKING CHANGE] The database tables must be deleted for:

new settings which support spot trading
for new indexing of candles table which now loads X times faster

[BREAKING CHANGE] Removed the driver for "Binance Inverse Futures" (known as coin-margined futures on Binance) from the list of supported exchanges for import candles.
[BREAKING CHANGE] Updated exchange names to be more similar to what they are on the exchange websites to avoid confusion.
[BREAKING CHANGE] Updated the required config object that is passed to the research.backtest() function based on the mentioned changes in this release. Removed the settlement_currency and added 'type'. Read the example on the documentation page for more details.
[BREAKING CHANGE] The alias self.capital has been removed to prevent confusion. Please use self.balance instead. Make sure to update your strategies accordingly.
[BREAKING CHANGE] The should_cancel() method has been renamed to should_cancel_entry() which is more convenient since it is only used for canceling entry orders. Make sure to update your strategies accordingly.
[NEW FEATURE] Added previous_qty property to the Position class.
[NEW FEATURE] Added the self.exchange_type, self.daily_balances properties to the Strategy class.
[NEW FEATURE] Added self.is_spot_trading and self.is_futures_trading properties to the Strategy class.
[IMPROVEMENT] The settlement currency is no longer fixed to USDT or USD. Instead, it is now the quote currency of the symbol you're trading. So it could be USD, USDT, BUSD, BTC (for spot markets), etc.
[IMPROVEMENT] The existence of plugins.py is now optional. It has been removed from new projects for the sake of simplicity. If you need to add a driver, you can do so by creating the file manually.
[IMPROVEMENT] The "Exchange type", "Leverage", and "Leverage Mode" are now displayed on backtests' result page so you know what the settings have been for the executed backtest.
[IMPROVEMENT] Minor improvements to the UI of the dashboard.
[FIX] Fixed a bug where generated CSV files in backtests were empty if the export_json option was disabled.

The breaking changes and why I made them

We're getting close to the stable release of Jesse version 1.0; and by doing so, I wanted to fix a few design mistakes that I initially made that caused confusion. I kept them so far, for the sake of limiting the breaking changes.

For instance, the naming of Binance Futures was a bit misleading because we don't support futures with expiration dates. We only support perpetual contracts. So I renamed it to Binance Perpetual Futures. Or now that we support spot trading, it made more sense to rename Binance to Binance Spot.

Another breaking change is the indexing of the database. I improved it, and now loading initial candles is significantly faster. In fact, in a test I ran, loading initial candles for 3 years of data took ~9 seconds before, but now it takes ~5 seconds. Since this was an indexing issue, the problem was growing as the size of your data increased. But now, it is fixed. Needless to say, this is just about the initial candle calls per each backtesting period and asset. If you try to run backtests on the same set of data, Jesse will use cache which loads significantly faster. In the case of the above, it takes ~1.5 seconds from the cache. This is not something new though. Caching has been implemented into Jesse for a while now.

Another breaking change that I made is related to the should_cancel() method which is used inside every strategy. It is now called should_cancel_entry() since it is only used for canceling entry orders. The previous name caused confusion as users thought they could use it for canceling orders everywhere in their strategies.

Update guide

Other than the typical update instructions that you have to usually follow, as this release includes a couple of breaking changes, other actions are required.

It might be easier to simply create a new Jesse project, and copy your strategies directory into the new project rather than updating the existing project. And of course, don't forget about updating the should_cancel() method's name to should_cancel_entry().

Here are the steps to update your project:

Either delete the plugins.py file from within your project, or in case you have defined your custom drivers, update the exchange names.
In case of using the live plugin, open your project's .env file, and change the exchange names for the API keys and secret keys to below:

# Testnet Binance Futures (http://testnet.binancefuture.com)
BINANCE_PERPETUAL_FUTURES_TESTNET_API_KEY=
BINANCE_PERPETUAL_FUTURES_TESTNET_API_SECRET=
# Binance Futures (https://www.binance.com/en/futures/btcusdt)
BINANCE_PERPETUAL_FUTURES_API_KEY=
BINANCE_PERPETUAL_FUTURES_API_SECRET=

# Testnet Bybit Perpetual (https://testnet.bybit.com/trade/usdt/BTCUSDT)
BYBIT_USDT_PERPETUAL_TESTNET_API_KEY=
BYBIT_USDT_PERPETUAL_TESTNET_API_SECRET=
# Bybit Perpetual (https://www.bybit.com/trade/usdt/BTCUSDT)
BYBIT_USDT_PERPETUAL_API_KEY=
BYBIT_USDT_PERPETUAL_API_SECRET=

# FTX Futures (https://ftx.com/markets/future)
FTX_PERPETUAL_FUTURES_API_KEY=
FTX_PERPETUAL_FUTURES_API_SECRET=
# leave empty if it's the main account and not a subaccount
FTX_PERPETUAL_FUTURES_SUBACCOUNT_NAME=

I usually write migrations for the modifications that I make to the database tables. But this release is an exception. You need to delete all the tables in the database (so that the next time you run Jesse, it'll create the new ones).

If you are using Jesse via Docker, just delete the postgres (which is the database) container:

# stop jesse's container in case they're still running them
docker-compose stop
# remove the Postgres container
docker rm postgres
# make sure Jesse's containers are up to date
docker-compose pull
# run jesse's containers again
docker-compose up

In case you're using Jesse via the native setup, use a database management software to delete all the tables in the database. TablePlus is the app that I recommend which supports macOS and Windows. Another option is to simply create another database and replace the keys in your project's .env file.

Differences between spot and futures modes

It's important to know the differences between spot and futures trading modes when writing your strategies. I added a new page to the documentation to explain the differences between them.

The simple problem in spot trading that's difficult to solve!

It's no secret that calculations of the spot mode are different than the futures mode. Spot trading seems easier at the first glance. But it's not. Not in our case at least. Here's what I've been struggling with, and my current solution for it:

Currently (via Jesse) you can set both the stop-loss and take-profit after a position is opened:

def on_open_position(self, order):
    self.stop_loss = self.position.qty, self.price - 10
    self.take_profit = self.position.qty, self.price + 10

And behind the scenes, Jesse will submit a LIMIT order for the take-profit order, and a STOP order for the stop-loss order to the exchange.

This works perfectly fine in a futures exchange. But in a spot exchange, each exchange out there is handling sell orders (which are reduce-only orders in the spot mode) differently. For example, on Bybit, you can submit both a LIMIT and STOP sell orders at the same time. Just like you would on a futures exchange. But on other exchanges such as Binance and FTX, you can't! You're only allowed to submit sell orders for as much as you have already bought.

So for example, if I buy (open a position) 1 BTC, then I can only submit orders as much as 1 BTC (in total). So I have to choose whether to submit my take-profit or stop-loss order. I can't submit both at the same time.

You might be thinking about OCO (One-Cancels-The-Other) orders on these exchanges which are usually supported on such exchanges. Building OCO orders into Jesse would in fact solve the issue in the case of the above example. However, one of Jesse's cool existing features is its support for partial entry and exits. Which OCO orders don't support. They are only useful if you are going in and out of positions all at once. For example, the below code won't work if I implemented OCO orders:

def on_open_position(self, order):
    self.stop_loss = self.position.qty, self.price - 10
    self.take_profit = self.position.qty, self.price + 10

My solution for it (so far)

In the backtesting, I'm implementing the simulation to be exactly how it is on Bybit Spot; which again, means you are allowed to submit two sell orders at the same time as long as one is a limit order and the other a stop order.

In the live trading mode, I'm implementing it with the help of queue orders. That means you don't have to change your strategy code to work in live mode. Jesse's smart enough to submit orders as much as the exchange accepts from it, and queue the rest (to submit them later when it is accepted by the targeted exchange).

I will explain this in the next release when I'm done with the implementation.

What's next?

My primary focus is to prepare the second part of adding support for spot trading. This time it'll be for live trading. Starting with Binance.com and FTX.com. Soon after I'll add Binance.us, FTX.us, and Bybit.com.

Thank you for your continued support,

— Saleh