diff --git a/appimage/README.md b/appimage/README.md index 2c8dd0d43..61ce64632 100644 --- a/appimage/README.md +++ b/appimage/README.md @@ -139,10 +139,15 @@ If OMEdit fails to launch, it's usually a missing Qt5 library. The builder autom To manually check: run `ldd eSim.AppDir/usr/bin/OMEdit.bin | grep "not found"` inside the build directory. -### Python Import Errors -If you see `ModuleNotFoundError` for PyQt5 or other Python packages: -- On **Arch**: Ensure `python-pyqt5` is installed (`sudo pacman -S python-pyqt5`) -- On **Fedora**: Ensure `python3-PyQt5` is installed (`sudo dnf install python3-PyQt5`) +### Qt / Wayland Display Issues +The AppImage automatically detects Wayland and X11 sessions and handles setting `QT_QPA_PLATFORM` and prioritizing bundled PyQt5 libraries. However, if you experience display initialization errors, you can force a platform by prepending the environment variable: +```bash +# Force X11 compatibility mode +QT_QPA_PLATFORM=xcb ./eSim-2.5.AppImage + +# Force Wayland native mode +QT_QPA_PLATFORM=wayland ./eSim-2.5.AppImage +``` ### Build Failures Common causes: @@ -150,24 +155,31 @@ Common causes: - **Missing disk space** (at least 5 GB required) - **Unsupported architecture** (only x86_64 is currently supported) +For details on past issues faced and how they were resolved, see: +- [appimage_issues.md](file:///home/ashu/Downloads/eSim_packagemanager/appimage/appimage_issues.md) — Documentation of error logs and root causes. +- [appimage_fix_implementation.md](file:///home/ashu/Downloads/eSim_packagemanager/appimage/appimage_fix_implementation.md) — Technical details of the python bundling and library isolation fixes. + --- ## 📁 Project Structure ``` eSim_Universal_packagemanager_linux/ -├── build-appimage.sh # Main build script (~7800 lines) -├── README.md # This file -└── build-eSim-AppImage/ # Created during build - ├── downloads/ # Downloaded tools & dependencies - ├── eSim.AppDir/ # AppImage filesystem - │ ├── AppRun # Entry point launcher +├── build-appimage.sh # Main build script (~7800 lines) +├── README.md # This file +├── appimage_issues.md # Documentation of errors and issues faced +├── appimage_fix_implementation.md # Technical details of the fixes implemented +└── build-eSim-AppImage/ # Created during build + ├── downloads/ # Downloaded tools & dependencies + ├── eSim.AppDir/ # AppImage filesystem + │ ├── AppRun # Entry point launcher (Wayland/X11 & Python wrapper) │ ├── usr/ - │ │ ├── bin/ # eSim, KiCad, OMEdit, NgSpice binaries - │ │ ├── lib/ # Bundled shared libraries - │ │ └── share/ # eSim source, resources, models + │ │ ├── bin/ # eSim, KiCad, OMEdit, NgSpice binaries + │ │ ├── lib/ # Bundled shared libraries + │ │ ├── python/ # Standalone bundled CPython 3.12 environment + │ │ └── share/ # eSim source, resources, models │ └── ... - └── eSim-2.5.AppImage # Final portable AppImage + └── eSim-2.5.AppImage # Final portable AppImage ``` --- diff --git a/appimage/appimage_fix_implementation.md b/appimage/appimage_fix_implementation.md new file mode 100644 index 000000000..caa1afbf7 --- /dev/null +++ b/appimage/appimage_fix_implementation.md @@ -0,0 +1,85 @@ +# eSim AppImage: Fixes and Implementation Approach + +This document explains the technical details, implementation plan, and approach used to resolve build-time and runtime issues for the eSim AppImage. + +--- + +## 1. Portable Python & Dependency Bundling +### Approach +To bypass host-system Python version mismatches and restrictions like PEP 668 (system-managed environments), we integrated a fully self-contained Python interpreter into the AppImage bundle. + +### Implementation +- During build time, the build script downloads a pre-compiled, portable CPython distribution (`python-build-standalone` version 3.12.1). +- It extracts this distribution to `$APPDIR/usr/python`. +- The build script installs all Python dependencies (including `PyQt5`, `hdlparse`, and others) directly into this bundled Python environment. +- At runtime, the `AppRun` launcher configures python env variables: + ```bash + export PYTHONHOME="${HERE}/usr/python" + export PYTHONPATH="${HERE}/usr/share/eSim/src" + export PATH="${HERE}/usr/python/bin:${PATH}" + ``` +- This completely eliminates dependency on the host's python interpreter or internet access for pip packages at runtime. + +--- + +## 2. Dynamic APT Package Detection +### Approach +To make the build script compatible across multiple generations of Debian and Ubuntu releases, we replaced hardcoded package lists with dynamic package detection. + +### Implementation +- We updated the `install_deps_apt` function in `build-appimage.sh` to query package managers: + ```bash + install_gdk_pixbuf() { + if apt-cache show libgdk-pixbuf-2.0-dev >/dev/null 2>&1; then + sudo apt-get install -y libgdk-pixbuf-2.0-dev + elif apt-cache show libgdk-pixbuf2.0-dev >/dev/null 2>&1; then + sudo apt-get install -y libgdk-pixbuf2.0-dev + fi + } + ``` +- This ensures the build process automatically adapts to changing package nomenclature upstream. + +--- + +## 3. Runtime Qt Platform Server Auto-Detection +### Approach +To support both modern Wayland-based desktops (like Ubuntu 22.04+ and 26.04 GNOME) and older X11-based desktops, we added a runtime display server detector to the main `AppRun` launcher. + +### Implementation +- Added detection code in the `AppRun` heredoc: + ```bash + if [ -z "${QT_QPA_PLATFORM}" ]; then + if [ -n "${WAYLAND_DISPLAY}" ] && [ "${XDG_SESSION_TYPE}" = "wayland" ]; then + export QT_QPA_PLATFORM=wayland + elif [ -n "${DISPLAY}" ]; then + export QT_QPA_PLATFORM=xcb + else + export QT_QPA_PLATFORM=xcb + fi + fi + ``` +- Pointed the Qt platform plugins engine to the bundled plugins directory from PyQt5: + ```bash + PYQT5_QT_PLUGINS="${HERE}/usr/python/lib/python3.12/site-packages/PyQt5/Qt5/plugins" + if [ -d "${PYQT5_QT_PLUGINS}" ]; then + export QT_PLUGIN_PATH="${PYQT5_QT_PLUGINS}:${HERE}/usr/lib/qt5/plugins:${QT_PLUGIN_PATH}" + else + export QT_PLUGIN_PATH="${HERE}/usr/lib/qt5/plugins:${QT_PLUGIN_PATH}" + fi + ``` + +--- + +## 4. Resolving Qt Library Version Conflicts +### Approach +We isolated the AppImage's Qt plugins from host system libraries by forcing the dynamic linker to prioritize the bundled PyQt5 Qt libraries. + +### Implementation +- Prepend the PyQt5 Qt5 library directory to `LD_LIBRARY_PATH` inside `AppRun`: + ```bash + _PYQT5_QT_LIB="${HERE}/usr/python/lib/python3.12/site-packages/PyQt5/Qt5/lib" + if [ -d "${_PYQT5_QT_LIB}" ]; then + export LD_LIBRARY_PATH="${_PYQT5_QT_LIB}:${LD_LIBRARY_PATH}" + fi + ``` +- This forces the system loader to resolve dependencies for `libqwayland-generic.so` and `libqxcb.so` against the bundled Qt libraries (`5.15.19`) instead of mixing them with host Qt libraries (like `5.15.18` on the host). diff --git a/appimage/appimage_issues.md b/appimage/appimage_issues.md new file mode 100644 index 000000000..06803279b --- /dev/null +++ b/appimage/appimage_issues.md @@ -0,0 +1,57 @@ +# eSim AppImage: Encountered Issues and Errors + +This document details the issues and error messages encountered during the build and runtime phases of the eSim AppImage on modern Linux environments (specifically tested on Ubuntu 26.04 Resolute with Wayland). + +--- + +## 1. Missing Python Packages (PyQt5 Dependency) +### Error Message +``` +Traceback (most recent call last): + File "/tmp/.mount_eSimXXXXXX/usr/bin/esim", line 3, in + from PyQt5.QtWidgets import QApplication +ModuleNotFoundError: No module named 'PyQt5' +``` + +### Context & Root Cause +Originally, the AppImage did not package its own Python environment or dependencies (like PyQt5). Instead, the `AppRun` entry point relied on the host system's Python interpreter (`/usr/bin/python3`) and attempted to run pip installation at runtime or assume that PyQt5 was installed on the host system. This failed because: +- Host systems may restrict ad-hoc package installations due to **PEP 668 (system-managed environments)**. +- Mismatched host Python versions caused binary package incompatibilities. +- The AppImage was not truly self-contained or portable. + +--- + +## 2. APT Package Installation Failures (Build-time Dependency) +### Error Message +``` +E: Unable to locate package libgdk-pixbuf2.0-dev +``` + +### Context & Root Cause +During the prerequisite installation stage on newer Ubuntu/Debian releases, the build script failed because package names had changed in the upstream repositories (e.g., `libgdk-pixbuf2.0-dev` became `libgdk-pixbuf-2.0-dev`). Hardcoded package names in the build script broke support on newer distributions. + +--- + +## 3. Qt Platform Plugin Initialization Failure (XCB) +### Error Message +``` +qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. +This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem. + +Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb. +``` + +### Context & Root Cause +The host environment was running a **Wayland** session (`XDG_SESSION_TYPE=wayland`). However, Qt5 defaults to the `xcb` (X11) platform plugin. Inside the AppImage container's execution sandbox, the host's X11/XCB libraries were either missing or misaligned with the loader's library path (`LD_LIBRARY_PATH`), causing the `libqxcb.so` plugin to fail to load. + +--- + +## 4. Incompatible Qt Libraries (Qt Version Mismatch) +### Error Message +``` +Cannot mix incompatible Qt library (5.15.19) with this library (5.15.18) +Aborted (core dumped) +``` + +### Context & Root Cause +When trying to run the Wayland platform plugin bundled with the PyQt5 wheel (`libqwayland-generic.so`), Qt attempted to resolve its dependencies. The bundled PyQt5 was compiled against Qt **5.15.19**, but the host system had Qt **5.15.18** installed. Because the host's `/usr/lib/x86_64-linux-gnu` took precedence in `LD_LIBRARY_PATH` over the PyQt5-specific directories, the dynamic linker loaded the host's older `libQt5Core.so.5`, causing a critical version conflict. diff --git a/appimage/build-appimage.sh b/appimage/build-appimage.sh old mode 100644 new mode 100755 index 05dcdb3ec..c523aab7a --- a/appimage/build-appimage.sh +++ b/appimage/build-appimage.sh @@ -10,7 +10,6 @@ BLUE='\033[0;34m'; CYAN='\033[0;36m'; MAGENTA='\033[0;35m' BOLD='\033[1m'; NC='\033[0m' print_header() { - clear echo -e "\n${BOLD}${CYAN}╔════════════════════════════════════════════════════════════╗${NC}" echo -e "${BOLD}${CYAN}║${NC} ${BOLD}eSim AppImage ${NC} ${BOLD}${CYAN}║${NC}" echo -e "${BOLD}${CYAN}╚════════════════════════════════════════════════════════════╝${NC}\n" @@ -290,7 +289,12 @@ install_deps_apt() { export TZ=UTC $SUDO apt-get update -qq 2>&1 | tail -3 || true - + + local pixbuf_pkg="libgdk-pixbuf2.0-0" + if ! apt-cache policy "$pixbuf_pkg" 2>/dev/null | grep -q "Candidate: [^(]"; then + pixbuf_pkg="libgdk-pixbuf-2.0-0" + fi + # Core dependencies - use DEBIAN_FRONTEND to prevent prompts DEBIAN_FRONTEND=noninteractive $SUDO apt-get install -y -qq \ ngspice verilator xterm \ @@ -298,7 +302,7 @@ install_deps_apt() { python3-numpy python3-scipy python3-lxml python3-matplotlib \ python3-pip python3-venv git build-essential \ libqt5gui5 libqt5widgets5 libqt5svg5 libqt5webenginewidgets5 \ - librsvg2-common libgdk-pixbuf2.0-0 gdk-pixbuf2.0-bin \ + librsvg2-common "$pixbuf_pkg" gdk-pixbuf2.0-bin \ adwaita-icon-theme hicolor-icon-theme file patchelf \ libfuse2 libglu1-mesa mesa-utils \ libboost-all-dev libglew2.2 libglm-dev \ @@ -670,6 +674,13 @@ download_tools() { -O eSim.zip 2>&1 | tail -2 fi + if [ ! -f python-3.12.tar.gz ]; then + progress "Downloading portable Python 3.12..." + wget -q --show-progress \ + "https://github.com/astral-sh/python-build-standalone/releases/download/20240107/cpython-3.12.1+20240107-x86_64-unknown-linux-gnu-install_only.tar.gz" \ + -O python-3.12.tar.gz 2>&1 | tail -2 + fi + # ═══ DOWNLOAD KICAD 6 APPIMAGE ═══ if [ ! -f kicad6.AppImage ] || [ ! -s kicad6.AppImage ]; then rm -f kicad6.AppImage 2>/dev/null || true @@ -2165,70 +2176,56 @@ NGHDLWRAPPER prepare_esim() { step 5 "Preparing eSim" cd "$DL" - progress "Extracting..." + progress "Extracting eSim..." unzip -qo eSim.zip - # Detect Python version dynamically - PYVER=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")') - PYVER_MAJOR=$(python3 -c 'import sys; print(sys.version_info.major)') - progress "Using Python $PYVER for virtual environment" - - # Create virtual environment - python3 -m venv "$BUILD/venv" || { - # Try alternative venv creation for some distros - progress "Trying alternative venv creation..." - python3 -m virtualenv "$BUILD/venv" 2>/dev/null || \ - virtualenv -p python3 "$BUILD/venv" 2>/dev/null || \ - die "Failed to create Python virtual environment" - } - source "$BUILD/venv/bin/activate" + # ═══ EXTRACT PORTABLE PYTHON 3.12 ═══ + progress "Extracting portable Python 3.12..." + rm -rf "$BUILD/python" 2>/dev/null || true + tar -xzf "$DL/python-3.12.tar.gz" -C "$BUILD" - # Upgrade pip and install dependencies - pip install -q --upgrade pip setuptools wheel 2>&1 | tail -2 + # Check if extraction was successful + if [ ! -d "$BUILD/python" ] || [ ! -x "$BUILD/python/bin/python3" ]; then + die "Failed to extract portable Python 3.12" + fi - # Install Python dependencies - these will be bundled in AppImage - # Using --no-cache-dir to ensure clean install - progress "Installing Python dependencies..." - pip install -q --no-cache-dir numpy scipy matplotlib lxml watchdog 2>&1 | tail -2 + PYVER="3.12" - # makerchip-app and sandpiper-saas are optional (for Makerchip integration) - pip install -q --no-cache-dir makerchip-app sandpiper-saas 2>&1 | tail -2 || \ - warn "makerchip-app/sandpiper-saas installation failed (optional)" + # Upgrade pip and install dependencies + progress "Upgrading pip/setuptools/wheel in bundled Python..." + "$BUILD/python/bin/python3" -m pip install -q --upgrade pip setuptools wheel 2>&1 | tail -2 + # Install Python dependencies - these will be bundled in AppImage + progress "Installing eSim Python dependencies..." + "$BUILD/python/bin/python3" -m pip install -q --no-cache-dir \ + numpy scipy matplotlib lxml watchdog PyQt5 PyQt5-sip PyQtWebEngine \ + makerchip-app sandpiper-saas 2>&1 | tail -2 + # Clone and install hdlparse if [ ! -d hdlparse ]; then git clone --quiet https://github.com/kevinpt/hdlparse.git fi # Find the correct site-packages directory - SITE_PACKAGES=$(python3 -c "import site; print(site.getsitepackages()[0])" 2>/dev/null || \ - echo "$BUILD/venv/lib/python${PYVER}/site-packages") + SITE_PACKAGES="$BUILD/python/lib/python3.12/site-packages" - cp -r hdlparse/hdlparse "$SITE_PACKAGES/" 2>/dev/null || \ - cp -r hdlparse/hdlparse "$BUILD/venv/lib/python${PYVER}/site-packages/" + cp -r hdlparse/hdlparse "$SITE_PACKAGES/" 2>/dev/null || true # Fix hdlparse for Python 3 compatibility MINILEXER="$SITE_PACKAGES/hdlparse/minilexer.py" - [ ! -f "$MINILEXER" ] && MINILEXER="$BUILD/venv/lib/python${PYVER}/site-packages/hdlparse/minilexer.py" - if [ -f "$MINILEXER" ]; then - # Fix Python 2 to Python 3 syntax sed -i 's/except \([A-Za-z_][A-Za-z0-9_]*\), \([A-Za-z_][A-Za-z0-9_]*\):/except (\1, \2):/' "$MINILEXER" sed -i 's/\.iteritems()/.items()/g; s/\.iterkeys()/.keys()/g; s/\.itervalues()/.values()/g' "$MINILEXER" fi # Create minilexer compatibility module - mkdir -p "$SITE_PACKAGES/minilexer" 2>/dev/null || \ - mkdir -p "$BUILD/venv/lib/python${PYVER}/site-packages/minilexer" + mkdir -p "$SITE_PACKAGES/minilexer" 2>/dev/null MINILEXER_INIT="$SITE_PACKAGES/minilexer/__init__.py" - [ ! -d "$(dirname "$MINILEXER_INIT")" ] && MINILEXER_INIT="$BUILD/venv/lib/python${PYVER}/site-packages/minilexer/__init__.py" - echo 'from hdlparse.minilexer import MiniLexer __all__ = ["MiniLexer"]' > "$MINILEXER_INIT" - deactivate - ok "eSim prepared with Python $PYVER" + ok "eSim prepared with portable Python $PYVER" } bundle_gtk_resources() { @@ -2690,7 +2687,7 @@ install_esim() { cd "$DL" - PYVER=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")') + PYVER="3.12" mkdir -p "$APPDIR/usr/share/eSim" "$APPDIR/usr/bin" "$APPDIR/usr/lib/python${PYVER}/site-packages" cp -r eSim-${ESIM_VERSION}/* "$APPDIR/usr/share/eSim/" @@ -6049,63 +6046,9 @@ PYCODE ok "eSim icons installed at multiple sizes" } - cp -r "$BUILD/venv/lib/python${PYVER}/site-packages/"* "$APPDIR/usr/lib/python${PYVER}/site-packages/" - - # ═══ REMOVE C-EXTENSION PACKAGES ═══ - # These packages contain compiled .so files that are Python-version-specific. - # They WILL NOT work on systems with different Python versions (e.g. 3.11 vs 3.12). - # Instead, we install them via pip at runtime to ~/.esim/python-packages/ - # This ensures the correct version is used for the host's Python interpreter. - progress "Removing Python-version-specific packages (will be installed at runtime)..." - - # Packages with C extensions that must match Python version - # NOTE: PIL is the actual package directory for Pillow - C_EXT_PACKAGES="numpy scipy matplotlib lxml contourpy kiwisolver pillow PIL fontTools" - - for pkg in $C_EXT_PACKAGES; do - # Remove from primary Python version - rm -rf "$APPDIR/usr/lib/python${PYVER}/site-packages/${pkg}" \ - "$APPDIR/usr/lib/python${PYVER}/site-packages/${pkg}-"*.dist-info \ - "$APPDIR/usr/lib/python${PYVER}/site-packages/${pkg,,}" \ - "$APPDIR/usr/lib/python${PYVER}/site-packages/${pkg,,}-"*.dist-info \ - 2>/dev/null || true - # Also remove from ALL Python version directories - for pyv in 3.8 3.9 3.10 3.11 3.12 3.13; do - rm -rf "$APPDIR/usr/lib/python${pyv}/site-packages/${pkg}" \ - "$APPDIR/usr/lib/python${pyv}/site-packages/${pkg}-"*.dist-info \ - "$APPDIR/usr/lib/python${pyv}/site-packages/${pkg,,}" \ - "$APPDIR/usr/lib/python${pyv}/site-packages/${pkg,,}-"*.dist-info \ - 2>/dev/null || true - done - done - - # Also remove the compiled .so files from any remaining packages - find "$APPDIR/usr/lib/python${PYVER}/site-packages" -name "*.cpython-*.so" -delete 2>/dev/null || true - - ok "Removed C-extension packages (will be installed via pip at first run)" - - # ═══ MULTI-DISTRO PYTHON: Bundle pure-Python packages for other versions ═══ - # Compiled .so extensions are ABI-specific and won't work across Python versions. - # Copy ONLY pure-Python packages (no .so files) to a fallback directory for - # distros with different Python versions (e.g. Ubuntu 22.04 has 3.10, 24.04 has 3.12) - for alt_py in 3.8 3.9 3.10 3.11 3.12 3.13; do - [ "$alt_py" = "$PYVER" ] && continue - alt_dir="$APPDIR/usr/lib/python${alt_py}/site-packages" - mkdir -p "$alt_dir" - # Copy only pure-Python packages (dirs without .so files) - for pkg_dir in "$APPDIR/usr/lib/python${PYVER}/site-packages/"*/; do - pkg_name=$(basename "$pkg_dir") - # Skip dist-info, __pycache__, and packages with compiled extensions - case "$pkg_name" in *.dist-info|*.egg-info|__pycache__) continue ;; esac - if ! find "$pkg_dir" -name '*.so' -print -quit 2>/dev/null | grep -q .; then - cp -r "$pkg_dir" "$alt_dir/" 2>/dev/null || true - fi - done - # Copy standalone .py files (argparse.py, etc) - cp "$APPDIR/usr/lib/python${PYVER}/site-packages/"*.py "$alt_dir/" 2>/dev/null || true - cp "$APPDIR/usr/lib/python${PYVER}/site-packages/"*.pth "$alt_dir/" 2>/dev/null || true - done - ok "Pure-Python packages replicated for multi-distro support" + progress "Bundling portable Python 3.12..." + cp -r "$BUILD/python" "$APPDIR/usr/" + ok "Portable Python 3.12 bundled successfully" mkdir -p "$APPDIR/usr/share/eSim/library/ngspicetoModelica" # Copy actual Mapping.json content (contains Sources, Devices, Components mappings) if [ -f "$APPDIR/usr/share/eSim/library/ngspicetoModelica/Mapping.json" ]; then @@ -6926,101 +6869,9 @@ _setup_compat_libs # we install them at first run to ~/.esim/python-packages/ # Pure-Python packages (hdlparse, minilexer) are bundled in the AppImage. -_ESIM_PYDIR="$HOME/.esim/python-packages" - _ensure_python_packages() { - # Check if critical packages are available (either system or our installed) - if python3 -c "import numpy, matplotlib, lxml" 2>/dev/null; then - # Already installed (system or ~/.esim), just add our dir to path - export PYTHONPATH="${_ESIM_PYDIR}:${PYTHONPATH}" - return 0 - fi - - # Need to install packages - echo "[eSim] Installing Python dependencies for $(python3 --version 2>&1)..." - echo "[eSim] This only happens once. Please wait..." - - mkdir -p "$_ESIM_PYDIR" - - # On Fedora, install matplotlib with Qt5 support from system first - if command -v dnf >/dev/null 2>&1; then - echo "[eSim] Installing system packages for Fedora..." - dnf install -y python3-matplotlib-qt5 python3-pyqt5 2>&1 | tail -1 || true - fi - - # Install required packages to user directory - # These are the packages with C extensions that must match Python version - local _pip_cmd="python3 -m pip install --target $_ESIM_PYDIR --upgrade --quiet" - - # Core scientific packages (do NOT include PyQt5 or matplotlib here on systems - # where system packages provide them with proper Qt5 support) - if command -v dnf >/dev/null 2>&1; then - # Fedora: skip matplotlib, use system python3-matplotlib-qt5 instead - $_pip_cmd numpy scipy lxml watchdog pillow pyqt5-sip 2>&1 | grep -v "already satisfied" | tail -3 - else - # Other distros: try to install matplotlib via pip - $_pip_cmd numpy scipy matplotlib lxml watchdog pillow pyqt5-sip 2>&1 | grep -v "already satisfied" | tail -3 - fi - - # Try pip-installing PyQt5 only if system doesn't have it - # (PyPI wheels only exist up to ~Python 3.12) - if ! python3 -c "import PyQt5" 2>/dev/null; then - # Try pip first (works on older Python versions) - if ! $_pip_cmd PyQt5 2>/dev/null; then - # pip failed — try installing system package automatically - echo "[eSim] PyQt5 not available via pip, trying system package..." - if command -v dnf >/dev/null 2>&1; then - dnf install -y python3-qt5 python3-qt5-base python3-matplotlib-qt5 2>&1 | tail -2 || true - elif command -v apt-get >/dev/null 2>&1; then - apt-get install -y -qq python3-pyqt5 python3-matplotlib 2>&1 | tail -2 || true - elif command -v pacman >/dev/null 2>&1; then - pacman -S --noconfirm python-pyqt5 python-matplotlib 2>&1 | tail -2 || true - elif command -v zypper >/dev/null 2>&1; then - zypper --non-interactive install python3-qt5 python3-matplotlib-qt5 2>&1 | tail -2 || true - fi - # Final check - if ! python3 -c "import PyQt5" 2>/dev/null; then - echo "[eSim] Warning: PyQt5 not found. Install system package:" - echo "[eSim] Debian/Ubuntu: sudo apt install python3-pyqt5 python3-matplotlib" - echo "[eSim] Fedora: sudo dnf install python3-qt5 python3-matplotlib-qt5" - echo "[eSim] Arch: sudo pacman -S python-pyqt5" - fi - fi - fi - - # Check installation success - if ! python3 -c "import sys; sys.path.insert(0, '$_ESIM_PYDIR'); import numpy" 2>/dev/null; then - echo "[eSim] Warning: Some Python packages failed to install." - echo "[eSim] Try: pip install --user numpy scipy matplotlib lxml watchdog" - else - echo "[eSim] ✓ Python packages installed successfully" - fi - - # Verify matplotlib has Qt5 backend support - if ! python3 -c "from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg" 2>/dev/null; then - echo "[eSim] Note: matplotlib Qt5 backend not available. Installing system packages..." - if command -v dnf >/dev/null 2>&1; then - # Fedora: must use system package, not pip - dnf install -y python3-matplotlib-qt5 2>&1 | tail -1 || true - # Verify again - if ! python3 -c "from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg" 2>/dev/null; then - echo "[eSim] ✓ Matplotlib Qt5 backend installed from system" - fi - elif command -v apt-get >/dev/null 2>&1; then - apt-get install -y -qq python3-matplotlib 2>&1 | tail -1 || true - elif command -v pacman >/dev/null 2>&1; then - pacman -S --noconfirm python-matplotlib 2>&1 | tail -1 || true - fi - - # Final verification - if ! python3 -c "from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg" 2>/dev/null; then - echo "[eSim] ⚠ Warning: matplotlib Qt5 backend still not available" - echo "[eSim] Try: sudo dnf install python3-matplotlib-qt5 (Fedora)" - echo "[eSim] Or: sudo apt install python3-matplotlib (Debian/Ubuntu)" - fi - fi - - export PYTHONPATH="${_ESIM_PYDIR}:${PYTHONPATH}" + # All packages are pre-bundled in the AppImage (using portable Python 3.12) + return 0 } _ensure_python_packages @@ -7476,7 +7327,7 @@ detect_lib_paths() { SYS_LIB_PATHS=$(detect_lib_paths) # Include ~/.local/bin for pipx-installed tools like makerchip -export PATH="${HERE}/usr/bin:${HOME}/.local/bin:${PATH}" +export PATH="${HERE}/usr/python/bin:${HERE}/usr/bin:${HOME}/.local/bin:${PATH}" # ═══ CHECK AND SETUP MAKERCHIP ═══ # Makerchip IDE is an external tool that must be installed via pipx @@ -7527,36 +7378,19 @@ else export LIBRARY_PATH="${SYS_LIB_PATHS}${LIBRARY_PATH}" fi -# PYTHONPATH with dynamic Python version detection -# Priority: 1. Runtime-installed packages (correct Python version) -# 2. eSim source code -# 3. Bundled pure-Python packages (hdlparse, etc.) -# 4. System site-packages (for PyQt5, etc. installed by distro package manager) -BUNDLED_SITE_PACKAGES="" -for pyver in "$PYTHON_VERSION" 3.14 3.13 3.12 3.11 3.10 3.9 3.8; do - if [ -d "${HERE}/usr/lib/python${pyver}/site-packages" ]; then - BUNDLED_SITE_PACKAGES="${HERE}/usr/lib/python${pyver}/site-packages" - break - fi -done - -# Detect system Python site-packages (needed for distro-installed PyQt5, etc.) -# Fedora/RHEL use /usr/lib64/pythonX.Y, Debian/Ubuntu use /usr/lib/python3/dist-packages -SYSTEM_SITE_PACKAGES="" -PYTHON_VERSION_DETECTED=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")' 2>/dev/null || echo "") -if [ -n "$PYTHON_VERSION_DETECTED" ]; then - for sp in "/usr/lib64/python${PYTHON_VERSION_DETECTED}/site-packages" \ - "/usr/lib/python${PYTHON_VERSION_DETECTED}/site-packages" \ - "/usr/lib/python3/dist-packages" \ - "/usr/lib/python${PYTHON_VERSION_DETECTED}/dist-packages"; do - if [ -d "$sp" ]; then - SYSTEM_SITE_PACKAGES="${SYSTEM_SITE_PACKAGES:+${SYSTEM_SITE_PACKAGES}:}${sp}" - fi - done +# Bundled Python Configuration +export PYTHONHOME="${HERE}/usr/python" +export PYTHONPATH="${HERE}/usr/share/eSim/src" + +# ═══ PREPEND PyQt5 Qt5 LIBS TO LD_LIBRARY_PATH ═══ +# The bundled PyQt5 ships its own Qt5 libraries. These MUST appear before any +# system Qt5 libs in LD_LIBRARY_PATH to prevent "Cannot mix incompatible Qt +# library" version conflicts (e.g. 5.15.19 bundled vs 5.15.18 system). +_PYQT5_QT_LIB="${HERE}/usr/python/lib/python3.12/site-packages/PyQt5/Qt5/lib" +if [ -d "${_PYQT5_QT_LIB}" ]; then + export LD_LIBRARY_PATH="${_PYQT5_QT_LIB}:${LD_LIBRARY_PATH}" fi -export PYTHONPATH="${_ESIM_PYDIR}:${HERE}/usr/share/eSim/src:${BUNDLED_SITE_PACKAGES}${SYSTEM_SITE_PACKAGES:+:${SYSTEM_SITE_PACKAGES}}" - # Generate loaders.cache at runtime - only for loaders that exist LOADER_DIR="${HERE}/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders" LOADER_CACHE="/tmp/esim-apprun-loaders-$$.cache" @@ -7588,6 +7422,32 @@ export GTK_THEME=Adwaita export GTK_MODULES="" export GTK3_MODULES="" export GTK_PATH="${HERE}/usr/lib/gtk-3.0" +# ═══ QT PLATFORM AUTO-DETECTION ═══ +# Auto-detect display server and set the correct Qt platform plugin. +# On Wayland sessions, Qt5 defaults to 'xcb' which requires X11/XCB libraries +# that may not be available in the AppImage container. Setting the correct +# platform plugin prevents the "Could not load the Qt platform plugin" crash. +if [ -z "${QT_QPA_PLATFORM}" ]; then + if [ -n "${WAYLAND_DISPLAY}" ] && [ "${XDG_SESSION_TYPE}" = "wayland" ]; then + export QT_QPA_PLATFORM=wayland + elif [ -n "${DISPLAY}" ]; then + export QT_QPA_PLATFORM=xcb + else + # No display server detected; try xcb as last resort + export QT_QPA_PLATFORM=xcb + fi +fi + +# ═══ QT PLUGIN PATH ═══ +# Point Qt to the bundled PyQt5 plugins so that the platform plugin (wayland +# or xcb) is always found from within the AppImage, regardless of what the +# host system has installed in its Qt directories. +PYQT5_QT_PLUGINS="${HERE}/usr/python/lib/python3.12/site-packages/PyQt5/Qt5/plugins" +if [ -d "${PYQT5_QT_PLUGINS}" ]; then + export QT_PLUGIN_PATH="${PYQT5_QT_PLUGINS}:${HERE}/usr/lib/qt5/plugins:${QT_PLUGIN_PATH}" +else + export QT_PLUGIN_PATH="${HERE}/usr/lib/qt5/plugins:${QT_PLUGIN_PATH}" +fi export QT_QPA_PLATFORMTHEME=gtk3 # ═══ NGSPICE AND NGHDL ENVIRONMENT VARIABLES ═══ diff --git a/flatpak/README.md b/flatpak/README.md index fbf5a46a1..b542ab30c 100644 --- a/flatpak/README.md +++ b/flatpak/README.md @@ -18,13 +18,14 @@ flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.f From the eSim project root: ```bash -flatpak-builder build flatpak/org.fossee.eSim.yml --install --user +flatpak-builder build flatpak/org.fossee.eSim.yml --install --user --force-clean ``` +*(The `--force-clean` flag is recommended to ensure a fresh build and correct offline download of dependencies).* Or from the flatpak directory: ```bash -flatpak-builder build org.fossee.eSim.yml --install --user +flatpak-builder build org.fossee.eSim.yml --install --user --force-clean ``` ## Running @@ -35,7 +36,7 @@ flatpak run org.fossee.eSim ## KiCad Integration -For schematic editing, install KiCad from Flathub (eSim will automatically use it when running as Flatpak): +For schematic editing, install KiCad from Flathub (eSim will automatically use `flatpak-spawn` to launch it from your host system when running as Flatpak): ```bash flatpak install flathub org.kicad.KiCad @@ -47,8 +48,8 @@ The Flatpak build focuses on core eSim functionality (schematic design with KiCa | Feature | Status | |---------|--------| -| NGHDL | Not included – use Ubuntu installer for Verilog/Modelica workflows | -| Makerchip | Not included | +| NGHDL | Not included – `hdlparse` dependency is incompatible with Python 3.12. Use Ubuntu installer for Verilog/Modelica workflows | +| Makerchip | Not included – Gracefully degrades via try/except for missing hdlparse dependency | | SKY130 PDK | Not bundled – use Ubuntu installer for mixed-signal with SKY130 | | xterm / gaw | Not included – external waveform viewer may need xterm on host | | NGHDL KiCad symbol creation | Limited – sandbox restricts writing to KiCad symbol directories | diff --git a/flatpak/README2.md b/flatpak/README2.md new file mode 100644 index 000000000..00c283388 --- /dev/null +++ b/flatpak/README2.md @@ -0,0 +1,55 @@ +# eSim Flatpak Configuration Details (README2) + +This document outlines the modifications made to successfully build the eSim Flatpak package, its known limitations, and the necessary commands to build and run it. + +## 🛠️ Changes Made for Flatpak Compatibility + +To ensure the eSim Flatpak package builds and installs correctly on modern Linux distributions, several adjustments were made to the `org.fossee.eSim.yml` manifest: + +1. **Fixed Ngspice URL and Checksum**: + - The previous `ngspice-45.2` source URL was broken (returned HTTP 404). + - Upgraded to **ngspice-46** (latest stable). + - Added `dest-filename: ngspice-46.tar.gz` because the SourceForge redirect URL strips the filename, which previously caused `flatpak-builder` to throw an "Unknown archive format" error. + +2. **Offline Python Dependency Installation**: + - *Problem*: The Flatpak build sandbox operates without network access, which causes live `pip install` commands to fail. + - *Solution*: All Python packages (and their transitive dependencies like `PyQt6-sip`, `PyQt6-Qt6`, `contourpy`, `cycler`, etc.) were pre-declared as `file` sources in the manifest. We used direct PyPI URLs and validated SHA256 hashes for each `.whl` file. + - Updated the build command to install completely offline: `pip3 install --no-index --find-links=wheels --prefix=/app`. + +3. **Removed `hdlparse` Dependency**: + - `hdlparse` uses a deprecated `setup.py` option (`use_2to3`) that was removed in newer versions of setuptools. It fundamentally fails to install on **Python 3.12** (which is shipped with the `org.freedesktop.Sdk//24.08` SDK). + - Consequently, `hdlparse` was omitted from the build to allow the core application to compile successfully. + +## ⚠️ Known Limitations + +Because of the restricted sandbox environment and library incompatibilities, the Flatpak version of eSim has the following limitations: + +1. **NGHDL is Unsupported**: Since `hdlparse` was removed (and due to the massive complexity of building the `ghdl` Ada compiler inside a Flatpak), the Verilog/Modelica digital modeling tool (NGHDL) will not work. +2. **KiCad Symbol Creation Restricted**: The sandbox restricts write access to system-level KiCad directories. +3. **No External Terminals**: Features that require an external host terminal (like `xterm` for `gaw` waveform viewing) are not included. + +*(Note: If you require full feature parity, especially for NGHDL or mixed-signal SKY130 workflows, it is highly recommended to use the native Ubuntu installer instead of Flatpak).* + +## 🚀 How to Build and Run + +### 1. Prerequisites +Ensure you have Flatpak and Flatpak-Builder installed on your system. You also **must** install the Flatpak version of KiCad, as eSim automatically looks for it to handle schematic design: + +```bash +flatpak install flathub org.kicad.KiCad +``` + +### 2. Build the Package +Navigate to the root directory of the eSim project and run the following command to build and install the package locally: + +```bash +flatpak-builder build flatpak/org.fossee.eSim.yml --install --user --force-clean +``` +*(The `--force-clean` flag ensures a fresh build without stale cache files).* + +### 3. Run eSim +Once the build is successful, launch the application with: + +```bash +flatpak run org.fossee.eSim +``` diff --git a/flatpak/org.fossee.eSim.yml b/flatpak/org.fossee.eSim.yml index 0cc7ec9fa..56a39b89f 100644 --- a/flatpak/org.fossee.eSim.yml +++ b/flatpak/org.fossee.eSim.yml @@ -20,12 +20,13 @@ finish-args: - --share=network - --socket=x11 - --socket=wayland + - --talk-name=org.freedesktop.Flatpak - --env=ESIM_FLATPAK=1 command: esim modules: - # Ngspice - circuit simulator (required for eSim simulations) + # Ngspice - circuit simulator (required for eSim simulations) - v46 - name: ngspice buildsystem: autotools config-opts: @@ -36,14 +37,101 @@ modules: - --with-ngshared sources: - type: archive - url: https://downloads.sourceforge.net/project/ngspice/ng-spice-rework/45.2/ngspice-45.2.tar.gz - sha256: ba8345f4c3774714c10f33d7da850d361cec7d14b3a295d0dc9fd96f7423812d + url: https://sourceforge.net/projects/ngspice/files/ng-spice-rework/46/ngspice-46.tar.gz/download + sha256: a0d1699af1940b06649276dcd6ff5a566c8c0cad01b2f7b5e99dedbb4d64c19b + dest-filename: ngspice-46.tar.gz - # eSim Python application (uses PyQt5) + # Python dependencies - all wheels pre-downloaded so pip can install offline. + # The flatpak build phase has no network access; network is only available to + # the running app via finish-args. All URLs and sha256s are from PyPI. + - name: python-deps + buildsystem: simple + build-commands: + - pip3 install --no-index --find-links=wheels --prefix=/app + PyQt6-Qt6 PyQt6-sip PyQt6 + numpy matplotlib contourpy cycler fonttools kiwisolver + packaging pillow pyparsing python-dateutil six + scipy watchdog psutil + sources: + # PyQt6 and its deps + - type: file + url: https://files.pythonhosted.org/packages/a8/07/21f7dc188e35b46631707f3b40ace5643a0e03a8e1e446854826d08a04ae/pyqt6_qt6-6.9.2-py3-none-manylinux_2_28_x86_64.whl + sha256: 9abfc0ee4a8293a6442128ae3f87f68e82e2a949d7b9caabd98c86ba5679ab48 + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/95/cb/116f9b328636765f3bce97d9e10ec041c54bbe92beb0617edb86c2b615c1/pyqt6_sip-13.11.1-cp312-cp312-manylinux1_x86_64.manylinux_2_5_x86_64.whl + sha256: 0df15849946cea969d3ff2b24b76149262b6044aea2c5403e4f70c24c973a4c8 + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/d5/78/92f3c46440a83ebe22ae614bd6792e7b052bcb58ff128f677f5662015184/pyqt6-6.9.1-cp39-abi3-manylinux_2_28_x86_64.whl + sha256: 37884df27f774e2e1c0c96fa41e817a222329b80ffc6241725b0dc8c110acb35 + dest: wheels + # numpy + - type: file + url: https://files.pythonhosted.org/packages/8c/3d/1e1db36cfd41f895d266b103df00ca5b3cbe965184df824dec5c08c6b803/numpy-2.2.6-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl + sha256: fd83c01228a688733f1ded5201c678f0c53ecc1006ffbc404db9f7a899ac6249 + dest: wheels + # matplotlib and its deps + - type: file + url: https://files.pythonhosted.org/packages/94/95/7f522393c88313336b20d70fc849555757b2e5febc22b83b3a3f0fd4bce9/matplotlib-3.11.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl + sha256: be5f93a1d21981bfb802ded0d77a0caa92d4342a47d45754fac77e314a506344 + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/a8/32/b8a1c8965e4f72482ff2d1ac2cd670ce0b542f203c8e1d34e7c3e6925da7/contourpy-1.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl + sha256: f26b383144cf2d2c29f01a1e8170f50dacf0eac02d64139dcd709a8ac4eb3cfe + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl + sha256: 85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30 + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/77/c7/2342da9830e3e9d4870305ca5d2091d2a83284f2953079b7bdd3b5e029d8/fonttools-4.63.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl + sha256: 58dc6bb86a78d782f00f9190ca02c119cf5bbe2807536e361e18d42019f877d8 + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/c4/13/680c54afe3e65767bed7ec1a15571e1a2f1257128733851ade24abcefbcc/kiwisolver-1.5.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl + sha256: bb5136fb5352d3f422df33f0c879a1b0c204004324150cc3b5e3c4f310c9049f + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl + sha256: 5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/e9/9e/c05e19657fd57841e476be1ab46c4d501bffbadbafdc31a6d665f8b737b6/pillow-12.2.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl + sha256: b86024e52a1b269467a802258c25521e6d742349d760728092e1bc2d135b4d76 + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl + sha256: 850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl + sha256: a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427 + dest: wheels + - type: file + url: https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl + sha256: 4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 + dest: wheels + # scipy + - type: file + url: https://files.pythonhosted.org/packages/79/2e/415119c9ab3e62249e18c2b082c07aff907a273741b3f8160414b0e9193c/scipy-1.16.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl + sha256: 72d1717fd3b5e6ec747327ce9bda32d5463f472c9dce9f54499e81fbd50245a1 + dest: wheels + # watchdog + - type: file + url: https://files.pythonhosted.org/packages/b5/e8/dbf020b4d98251a9860752a094d09a65e1b436ad181faf929983f697048f/watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl + sha256: 20ffe5b202af80ab4266dcd3e91aae72bf2da48c0d33bdb15c66658e685e94e2 + dest: wheels + # psutil + - type: file + url: https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl + sha256: 076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9 + dest: wheels + + # eSim Python application - name: esim buildsystem: simple build-commands: - - pip3 install --no-build-isolation --prefix=/app PyQt5 numpy matplotlib scipy watchdog hdlparse psutil - mkdir -p /app/share/esim - cp -r src library images /app/share/esim/ - cp VERSION LICENSE /app/share/esim/ 2>/dev/null || true diff --git a/src/maker/Maker.py b/src/maker/Maker.py index 21bdc4bb0..6f5bd4673 100644 --- a/src/maker/Maker.py +++ b/src/maker/Maker.py @@ -27,7 +27,10 @@ # ========================================================================= # importing the files and libraries -import hdlparse.verilog_parser as vlog +try: + import hdlparse.verilog_parser as vlog +except ImportError: + vlog = None from PyQt6 import QtCore, QtWidgets from PyQt6.QtCore import QThread, pyqtSignal from configuration.Appconfig import Appconfig diff --git a/src/maker/ModelGeneration.py b/src/maker/ModelGeneration.py index 3950eb20f..ea945bb74 100644 --- a/src/maker/ModelGeneration.py +++ b/src/maker/ModelGeneration.py @@ -34,7 +34,10 @@ from configuration import Appconfig from . import createkicad -import hdlparse.verilog_parser as vlog +try: + import hdlparse.verilog_parser as vlog +except ImportError: + vlog = None class ModelGeneration(QtWidgets.QWidget): diff --git a/src/projManagement/Kicad.py b/src/projManagement/Kicad.py index 679f86e55..73dec956f 100644 --- a/src/projManagement/Kicad.py +++ b/src/projManagement/Kicad.py @@ -95,7 +95,7 @@ def openSchematic(self): # (install: flatpak install flathub org.kicad.KiCad) if os.environ.get('ESIM_FLATPAK') == '1': self.cmd = ( - "flatpak run --command=eeschema org.kicad.KiCad " + + "flatpak-spawn --host flatpak run --command=eeschema org.kicad.KiCad " + schematic_file ) else: