BossaIntradayDownloader

Lean, resumable downloader for BOSSA.pl intraday tick data (GPW universe).

Single entrypoint: bash scripts/bossa_run.sh — orchestrates session keepalive, auth validation, and high-concurrency async download with a live terminal TUI.

Requirements

Requirement	Details
OS	Linux / WSL (Playwright + Chromium)
Python	3.10+ (repo uses 3.12 in .venv)
Python deps	aiohttp, playwright, optional uvloop
Browser	Chromium, installed via Playwright

Install

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
playwright install chromium

If Playwright / Chromium fails to start due to missing system libraries, install the dependencies for your distro using Playwright's recommended method.

Usage

First run — manual login (recommended, handles MFA)

bash scripts/bossa_run.sh --manual

Opens a headed Chromium profile (.pw-bossa-profile/). Log in to your BOSSA account, go to Notowania → Intraday, complete MFA if prompted, then return to the terminal and press Enter. The runner switches to headless keepalive and starts downloading.

Subsequent runs — automatic

bash scripts/bossa_run.sh

If the saved browser session is still valid, no manual steps are needed.

Auto-login (fully unattended)

Create a .env file in the repo root (git-ignored):

BOSSA_AUTO_LOGIN=1
BOSSA_USERNAME=your_login_id
BOSSA_PASSWORD=your_password

With auto-login enabled, the keepalive process automatically fills credentials and handles the MFA mobile push flow (sends a push notification to your bossaMobile app, beeps, and waits).

Custom date range / subset of symbols

START_DATE=2024-01-01 END_DATE=2024-12-31 bash scripts/bossa_run.sh

To download only specific symbols, create a text file (one symbol per line, # comments allowed):

SYMBOLS_FILE=my_symbols.txt bash scripts/bossa_run.sh

Re-download existing files

OVERWRITE=1 bash scripts/bossa_run.sh

Direct Python invocation (skip keepalive / auth phases)

If you already have a valid cookie in bossa_cookie.txt:

. .venv/bin/activate
python -m bossa_downloader.universe \
  --instruments-file all_instruments_list.json \
  --start 2024-01-01 --end 2024-12-31 \
  --cookie-file bossa_cookie.txt \
  --out-root data/bossa_universe_ticks \
  --concurrency 256

CLI flags

Flag	Effect
--manual	Headed browser for manual login (best for MFA / first run)
--headed	Force headed keepalive (debugging)
--no-smoke	Skip the download smoke test before starting

Configuration (environment variables)

All variables have sensible defaults. Override by prefixing the command: CONCURRENCY=128 START_DATE=2024-01-01 bash scripts/bossa_run.sh

Core settings

Variable	Default	Description
OUT_ROOT	data/bossa_universe_ticks	Output directory for downloaded files
START_DATE	2020-04-01	First date to download (YYYY-MM-DD)
END_DATE	(today)	Last date to download (YYYY-MM-DD)
CONCURRENCY	256	Max parallel HTTP requests
COOKIE_FILE	bossa_cookie.txt	Path to session cookie file (auto-managed by keepalive)
INSTRUMENTS_FILE	all_instruments_list.json	BOSSA instruments JSON (r_[].nazwa_sk = symbols)
SYMBOLS_FILE	(empty = all)	Optional: text file with symbols to download (one per line)
OVERWRITE	0	Set to 1 to re-download existing files

Keepalive / auth

Variable	Default	Description
KEEPALIVE_INTERVAL_S	45	Seconds between session refresh cycles
KEEPALIVE_HEADLESS	1	1 = headless Chromium, 0 = headed (visible window)
AUTH_WAIT_TIMEOUT_S	300	Max seconds to wait for valid auth before aborting
AUTH_RETRY_MAX	5	Max download retries on auth expiry
AUTH_RETRY_DELAY_S	5	Seconds to wait before retrying after auth failure
SMOKE_ENABLE	1	1 = run a download smoke test before bulk download
MANUAL_LOGIN	0	1 = manual login mode (or use --manual flag)
MANUAL_LOGIN_TIMEOUT_S	600	Max seconds to wait for manual login completion

Download tuning

Variable	Default	Description
TIMEOUT_S	120	HTTP request timeout (seconds)
AIOHTTP_DNS_TTL_S	60	DNS cache TTL for aiohttp
AIO_WRITE_WORKERS	2	Disk write thread pool size

Auto-login (.env file)

Variable	Default	Description
BOSSA_AUTO_LOGIN	0	1 = enable automatic credential submission
BOSSA_USERNAME	(required)	BOSSA login ID
BOSSA_PASSWORD	(required)	BOSSA password
BOSSA_MFA_WAIT_S	300	Max seconds to wait for MFA mobile confirmation
BOSSA_LOGIN_DEBUG	0	1 = save login screenshots to data/login_debug/
BOSSA_USER_SELECTOR	(auto)	Override CSS selector for username field
BOSSA_PASS_SELECTOR	(auto)	Override CSS selector for password field
BOSSA_SUBMIT_SELECTOR	(auto)	Override CSS selector for submit button

Output structure

data/bossa_universe_ticks/
├── manifest.json                    # Run configuration snapshot
├── run_results.json                 # Final stats from last run
├── _meta/
│   ├── progress_state.json          # Live progress (read by TUI)
│   ├── download.log                 # Downloader stdout/stderr
│   └── .unauthorized                # Symbols returning 401 (skipped)
├── KGH/
│   ├── KGH_2024-01-02.csv.gz       # Tick data (actually ZIP, see below)
│   ├── KGH_2024-01-03.csv.gz
│   └── .nodata                      # Dates with no data (cached)
├── PKO/
│   └── ...
└── ...

Note: Despite the .csv.gz suffix, downloaded files are ZIP archives (magic PK..). Use unzip, not gunzip:

unzip -l data/bossa_universe_ticks/KGH/KGH_2025-01-02.csv.gz
unzip -p data/bossa_universe_ticks/KGH/KGH_2025-01-02.csv.gz | head

Resumability

• Existing files — already-downloaded .csv.gz files are skipped automatically.
• No-data cache — dates that returned empty/404/422 are cached in <SYMBOL>/.nodata and skipped on re-runs.
• Unauthorized symbols — symbols returning 401 (while the session is valid) are persisted to _meta/.unauthorized and skipped. Delete this file to retry them after changing your BOSSA subscription.

Architecture

scripts/bossa_run.sh          # Orchestrator (single entrypoint)
├── tools/bossa/keepalive.sh  # Cleanup + launch keepalive
│   └── tools/bossa/keepalive_playwright.py  # Browser session manager
├── scripts/bossa_tui.py      # Live terminal UI (reads progress_state.json)
└── bossa_downloader/
    └── universe.py            # Async downloader core (aiohttp)

Workflow phases:

1. Keepalive — launches Playwright Chromium to keep the BOSSA session alive, periodically refreshing cookies to bossa_cookie.txt.
2. Auth probe + smoke test — validates the cookie works against the BOSSA API before starting bulk downloads.
3. Download — async aiohttp-based downloader with configurable concurrency. Progress is written to progress_state.json for the TUI. On auth expiry, the downloader exits with rc=2 and the runner retries (up to AUTH_RETRY_MAX).

Log files

File	Content
data/keepalive.log	Keepalive process output (login, cookie refresh, errors)
data/bossa_universe_ticks/_meta/download.log	Downloader process output (progress, errors)
data/login_debug/*.png	Login screenshots (when BOSSA_LOGIN_DEBUG=1)
bossa_cookie_status.json	Machine-readable keepalive status (used by TUI)

Troubleshooting

"Auth timeout after 300s" / "Cannot proceed without valid auth"

1. Run the manual flow: bash scripts/bossa_run.sh --manual
2. Make sure you navigated to Notowania → Intraday before pressing Enter.
3. Inspect keepalive log: tail -50 data/keepalive.log

Keepalive fails to start / "Failed to launch persistent browser context"

Common causes:

• Another Chromium/Playwright instance is using .pw-bossa-profile
• Stale Chromium lock files in .pw-bossa-profile/

Fix:

# Close all Playwright Chromium windows, then:
bash scripts/bossa_run.sh --manual
# If it still fails, delete the profile (you will need to log in again):
rm -rf .pw-bossa-profile/

Many "no data" results (especially for early START_DATE)

Expected. Most instruments have no intraday ticks for most historical trading days. The downloader caches these in <SYMBOL>/.nodata so subsequent runs are fast.

Many symbols skipped as unauthorized

If a symbol returns 401 while the session is valid, it means it's not in your BOSSA subscription. These are saved to data/bossa_universe_ticks/_meta/.unauthorized.

To retry after a subscription change:

rm data/bossa_universe_ticks/_meta/.unauthorized
bash scripts/bossa_run.sh

Download seems stuck / stale TUI

Check if the downloader process is running:

ps aux | grep "bossa_downloader.universe"

Check recent download activity:

tail -20 data/bossa_universe_ticks/_meta/download.log
cat data/bossa_universe_ticks/_meta/progress_state.json | python3 -m json.tool