Add storage examples for databases and s3

README.md

@@ -14,6 +14,7 @@ For full platform documentation, see [docs.prokube.ai](https://docs.prokube.ai/)
 ├── pipelines        # Kubeflow Pipelines examples
 ├── rstudio          # RStudio examples
 ├── serving          # model serving examples (KServe, vLLM, shadow deployments)
+├── storage          # object storage and database access examples
 ```
 
 ## Note about storage
@@ -35,6 +36,8 @@ Many S3 libraries use environment variables for their configuration — those ar
 `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `S3_ENDPOINT`. They are likely already
 available in your environment. You can also ask your admin about them.
 
+See `storage/s3` for S3 examples and `storage/database-access` for relational database examples in Python and R.
+
 ## Platform compatibility
 
 Some examples require a minimum prokube platform version. If an example is not listed, no specific version requirement is known.

storage/README.md

@@ -0,0 +1,8 @@
+# Storage Examples
+
+This directory contains examples for accessing storage services from prokube labs.
+
+- `s3`: read and write data with S3-compatible object storage.
+- `database-access`: connect to PostgreSQL, MariaDB/MySQL, and SQL Server-compatible databases.
+
+The examples are available for Python notebooks and RStudio. They use the packages already provided by the pk notebook and pk RStudio images.

storage/database-access/README.md

@@ -0,0 +1,38 @@
+# Database Access
+
+These examples show how to connect from the pk notebook and pk RStudio images to common relational databases.
+
+The pk notebook and pk RStudio images already include the required ODBC drivers and language packages. Each example uses the Iris dataset, creates a table named `iris_example`, writes the data, reads a few rows back, and closes the connection.
+
+## Examples
+
+- PostgreSQL: `python/postgresql.ipynb`, `r/postgresql.R`
+- MariaDB/MySQL: `python/mariadb.ipynb`, `r/mariadb.R`
+- Microsoft SQL Server: `python/sqlserver.ipynb`, `r/sqlserver.R`
+
+Edit only the connection values at the top of the example you want to run:
+
+- `host`
+- `port`
+- `database`
+- `user`
+
+The examples prompt for the password.
+
+## Drivers
+
+The Python notebooks use these database drivers:
+
+- PostgreSQL: `psycopg2`
+- MariaDB/MySQL: `pyodbc` with `MariaDB Unicode`
+- Microsoft SQL Server: `pyodbc` with `FreeTDS`
+
+The R scripts use `DBI` and `odbc` with the same ODBC drivers.
+
+## Microsoft SQL Server
+
+For Microsoft SQL servers you first need to find out which port the server is using. Open a terminal in RStudio and run:
+
+```sh
+tsql -L -H <server-domain>
+```

storage/database-access/python/duckdb.ipynb

@@ -0,0 +1,102 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "250f9140",
+   "metadata": {},
+   "source": [
+    "## DuckDB\n",
+    "DuckDB can read files from S3-backed storage through its `httpfs` extension. In managed Labs, S3-compatible endpoint and credential environment variables are usually provided by the workspace configuration."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6b98f465",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Install duckdb\n",
+    "!pip install duckdb"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0ba83c11",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import duckdb\n",
+    "import os \n",
+    "\n",
+    "conn = duckdb.connect()\n",
+    "conn.execute(\"INSTALL httpfs\")\n",
+    "conn.execute(\"LOAD httpfs\")\n",
+    "conn.execute(\"SET s3_url_style='path'\")\n",
+    "conn.execute(f\"SET s3_endpoint='{os.environ['S3_ENDPOINT']}'\")\n",
+    "conn.execute(f\"SET s3_use_ssl = {'false' if os.environ['S3_USE_HTTPS'] == '0' else 'true'}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d990fd48",
+   "metadata": {},
+   "source": [
+    "To check whether DuckDB can discover credentials in the current environment, run `CALL load_aws_credentials()` after loading the `httpfs` extension."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8d7c6b5a",
+   "metadata": {},
+   "source": [
+    "## Iris dataset in S3 Parquet\n",
+    "\n",
+    "Write the Iris dataset to an S3-backed Parquet file and read it back with DuckDB."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6f5e4d3c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.datasets import load_iris\n",
+    "\n",
+    "bucket = \"<bucketname>\"\n",
+    "key = \"storage-examples/iris.parquet\"\n",
+    "\n",
+    "if not bucket or bucket == \"<bucketname>\":\n",
+    "    raise ValueError(\"Set bucket to your S3 bucket name.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2b1a0f9e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "iris = load_iris(as_frame=True).frame\n",
+    "s3_path = f\"s3://{bucket}/{key}\"\n",
+    "\n",
+    "conn.register(\"iris\", iris)\n",
+    "conn.execute(\n",
+    "    f\"COPY iris TO '{s3_path}' (FORMAT PARQUET, OVERWRITE_OR_IGNORE TRUE)\"\n",
+    ")\n",
+    "\n",
+    "read_back = conn.sql(f\"SELECT * FROM read_parquet('{s3_path}')\").df()\n",
+    "read_back.head()"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

storage/database-access/python/mariadb.ipynb

@@ -0,0 +1,163 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "812eb55e",
+   "metadata": {},
+   "source": [
+    "# MariaDB/MySQL Access\n",
+    "\n",
+    "Write the Iris dataset to MariaDB or MySQL and read it back."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "db21f0c0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import getpass\n",
+    "\n",
+    "import pandas as pd\n",
+    "\n",
+    "from sklearn.datasets import load_iris"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "90a6af60",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "host = \"<server-domain>\"\n",
+    "port = 3306\n",
+    "database = \"<database>\"\n",
+    "user = \"<user>\"\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3100e7e2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = load_iris(as_frame=True)\n",
+    "df.frame.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "502ba2cb",
+   "metadata": {},
+   "source": [
+    "## SQLAlchemy \n",
+    "[SQLAlchemy](https://www.sqlalchemy.org/) is a Python library that provides a database abstraction layer so you can connect to different SQL databases.\n",
+    "It works well with `pandas.to_sql()` and `pandas.read_sql()` and can be used with different backends.  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a328ba04",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sqlalchemy import URL, create_engine\n",
+    "\n",
+    "engine = create_engine(\n",
+    "    URL.create(\n",
+    "        \"mysql+pyodbc\",\n",
+    "        username=user,\n",
+    "        password=getpass.getpass(\"Database password: \"),\n",
+    "        host=host,\n",
+    "        port=port,\n",
+    "        database=database,\n",
+    "        query={\"driver\": \"MariaDB Unicode\"},\n",
+    "    )\n",
+    ")\n",
+    "\n",
+    "df.frame.to_sql(\"iris_example\", engine, if_exists=\"replace\", index=False)\n",
+    "\n",
+    "pd.read_sql(\"SELECT * FROM iris_example\", engine).head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0ebb300",
+   "metadata": {},
+   "source": [
+    "## pyodbc cursor\n",
+    "\n",
+    "Direct `pyodbc` cursor calls are more verbose, but they show the underlying DB-API flow: open a connection, create a cursor, execute SQL, commit, fetch rows, and close the connection.\n",
+    "\n",
+    "Documentation: [pyodbc](https://github.com/mkleehammer/pyodbc/wiki)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f34c3be1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pyodbc\n",
+    "\n",
+    "conn = pyodbc.connect(\n",
+    "    \"DRIVER={MariaDB Unicode};\"\n",
+    "    f\"SERVER={host};\"\n",
+    "    f\"PORT={port};\"\n",
+    "    f\"DATABASE={database};\"\n",
+    "    f\"UID={user};\"\n",
+    "    f\"PWD={getpass.getpass('Database password: ')}\"\n",
+    ")\n",
+    "\n",
+    "try:\n",
+    "    cursor = conn.cursor()\n",
+    "    cursor.execute(\"DROP TABLE IF EXISTS iris_example_cursor\")\n",
+    "    cursor.execute(\"\"\"\n",
+    "        CREATE TABLE iris_example_cursor (\n",
+    "            `sepal length (cm)` DOUBLE,\n",
+    "            `sepal width (cm)` DOUBLE,\n",
+    "            `petal length (cm)` DOUBLE,\n",
+    "            `petal width (cm)` DOUBLE,\n",
+    "            target INTEGER\n",
+    "        )\n",
+    "    \"\"\")\n",
+    "    cursor.executemany(\"\"\"\n",
+    "        INSERT INTO iris_example_cursor (\n",
+    "            `sepal length (cm)`,\n",
+    "            `sepal width (cm)`,\n",
+    "            `petal length (cm)`,\n",
+    "            `petal width (cm)`,\n",
+    "            target\n",
+    "        )\n",
+    "        VALUES (?, ?, ?, ?, ?)\n",
+    "    \"\"\", df.frame.itertuples(index=False, name=None))\n",
+    "    conn.commit()\n",
+    "\n",
+    "    cursor.execute(\"SELECT * FROM iris_example_cursor LIMIT 5\")\n",
+    "    rows = cursor.fetchall()\n",
+    "    columns = [column[0] for column in cursor.description]\n",
+    "    display(pd.DataFrame.from_records(rows, columns=columns))\n",
+    "finally:\n",
+    "    conn.close()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}