Data extraction via MT5 api¶
Each trading symbol has its own process for the wine mt5 api. This is so that in production, we're able to reduce latency (at the cost of higher memory consumption) by pulling from one or more brokers in parallel.
PREREQUISITE - start mt5 apis¶
Before running this notebooks, first launch the mt5 apis from the docker terminal using the command:
releat launch-all-mt5-apis
Alternatively, you can run it from your local terminal and execute on your docker container, replace <container-name>
with the name of the container, which should either be releat
or releat-dc
depending on how you set it up:
docker exec -it <container-name> releat launch-all-mt5-apis
Launching wine processes from jupyter notebooks is unstable and mostly fails to start
from releat.utils.configs.constants import mt5_api_port_map
from concurrent.futures import ThreadPoolExecutor
from releat.workflows.service_manager import kill_processes, get_pids, stop_mt5
from releat.utils.logging import get_logger
from releat.data.extractor import download_tick_data
import logging
from datetime import datetime
logger = get_logger(__name__, log_level=logging.INFO)
# start date - datetime must be in this format
dt0 = datetime.strptime("2023-09-06 10:00:00.000","%Y-%m-%d %H:%M:%S.%f")
# end date - excludes boundary
dt1 = datetime.strptime("2023-09-06 10:00:10.001","%Y-%m-%d %H:%M:%S.%f")
# data_mode - either demo or live
data_mode = "demo"
# check api - whether to check the connection, if failed, it will try to initialize the
# connection. checking is skipped at inference
check_api = False
dl_args = []
for broker, port_map in mt5_api_port_map.items():
for symbol, port in port_map.items():
# general is the port used for other interactions with mt5, i.e. order and
# getting position
if symbol!='general':
dl_arg = [broker,symbol,dt0,dt1,data_mode,check_api]
dl_args.append(dl_arg)
print("An example of input list to the download_tick_data function:")
print(dl_args[0])
An example of input list to the download_tick_data function: ['metaquotes', 'EURUSD', datetime.datetime(2023, 9, 6, 10, 0), datetime.datetime(2023, 9, 6, 10, 0, 10, 1000), 'demo', False]
# show example output of function
download_tick_data(*dl_args[1])
2023-11-04 10:35:44,096 INFO releat.data.extractor | metaquotes AUDJPY 2023-09-06 10:00:00.000000 >> 2023-09-06 10:00:10.001000 65 ticks
{'broker': 'metaquotes', 'symbol': 'AUDJPY', 'tick_df': bid ask last volume time_msc flags volume_real 0 94.084 94.093 0.0 0 2023-09-06 10:00:00.016 130 0.0 1 94.083 94.092 0.0 0 2023-09-06 10:00:00.063 134 0.0 2 94.085 94.093 0.0 0 2023-09-06 10:00:00.116 134 0.0 3 94.085 94.094 0.0 0 2023-09-06 10:00:00.151 4 0.0 4 94.087 94.095 0.0 0 2023-09-06 10:00:00.174 134 0.0 .. ... ... ... ... ... ... ... 60 94.071 94.079 0.0 0 2023-09-06 10:00:09.196 130 0.0 61 94.071 94.080 0.0 0 2023-09-06 10:00:09.345 4 0.0 62 94.074 94.082 0.0 0 2023-09-06 10:00:09.628 134 0.0 63 94.074 94.083 0.0 0 2023-09-06 10:00:09.677 4 0.0 64 94.075 94.084 0.0 0 2023-09-06 10:00:09.745 134 0.0 [65 rows x 7 columns]}
pool = ThreadPoolExecutor(len(dl_args))
Compare data extraction speed¶
When run in parallel, the time taken to download data is at least 2x as fast as compared to in sequence. Note results will vary depending on internet speed and whether results are cached
%%timeit
for dl_arg in dl_args:
download_tick_data(*dl_arg)
%%timeit
list(pool.map(lambda p: download_tick_data(*p), dl_args))
Clean processes by deleting MT5 instances and apis¶
# kill mt5
stop_mt5()
2023-11-04 10:35:52,946 INFO releat.workflows.service_manager | process id: 728095 killed 2023-11-04 10:35:52,947 INFO releat.workflows.service_manager | process id: 728698 killed 2023-11-04 10:35:52,948 INFO releat.workflows.service_manager | MetaTrader5 stopped: process ids [728095, 728698] killed
# kill mt5 api process ids
pids = get_pids("wineserver")
kill_processes(pids)
print(f"mt5 apis stopped - process ids {pids} killed")
# kill wine processes
pids = get_pids("python.exe")
kill_processes(pids)
2023-11-04 10:35:55,953 INFO releat.workflows.service_manager | process id: 727954 killed mt5 apis stopped - process ids [727954] killed 2023-11-04 10:35:55,962 INFO releat.workflows.service_manager | process id: 728002 killed 2023-11-04 10:35:55,965 INFO releat.workflows.service_manager | process id: 728057 killed 2023-11-04 10:35:55,967 INFO releat.workflows.service_manager | process id: 728152 killed 2023-11-04 10:35:55,968 INFO releat.workflows.service_manager | process id: 728248 killed 2023-11-04 10:35:55,970 INFO releat.workflows.service_manager | process id: 728283 killed 2023-11-04 10:35:55,971 INFO releat.workflows.service_manager | process id: 728343 killed 2023-11-04 10:35:55,972 INFO releat.workflows.service_manager | process id: 728380 killed 2023-11-04 10:35:55,973 INFO releat.workflows.service_manager | process id: 728444 killed 2023-11-04 10:35:55,974 INFO releat.workflows.service_manager | process id: 728485 killed 2023-11-04 10:35:55,974 INFO releat.workflows.service_manager | process id: 728558 killed 2023-11-04 10:35:55,975 INFO releat.workflows.service_manager | process id: 728595 killed 2023-11-04 10:35:55,976 INFO releat.workflows.service_manager | process id: 728665 killed 2023-11-04 10:35:55,979 INFO releat.workflows.service_manager | process id: 728755 killed 2023-11-04 10:35:55,981 INFO releat.workflows.service_manager | process id: 728844 killed 2023-11-04 10:35:55,983 INFO releat.workflows.service_manager | process id: 728879 killed 2023-11-04 10:35:55,984 INFO releat.workflows.service_manager | process id: 728960 killed 2023-11-04 10:35:55,986 INFO releat.workflows.service_manager | process id: 728991 killed 2023-11-04 10:35:55,987 INFO releat.workflows.service_manager | process id: 729070 killed 2023-11-04 10:35:55,987 INFO releat.workflows.service_manager | process id: 729105 killed 2023-11-04 10:35:55,988 INFO releat.workflows.service_manager | process id: 729186 already killed