diff --git a/docs/porting_guide.rst b/docs/porting_guide.rst index 633e129..82c69fc 100644 --- a/docs/porting_guide.rst +++ b/docs/porting_guide.rst @@ -4,6 +4,253 @@ Porting GuideLine ================= +******** +Hardware +******** + +The porting of OpenAMP to a new :doc:`multicore system <../openamp/overview>` requires +some hardware for inter-processor communication. + +- Shared memory for + + - the :ref:`RPMsg` buffer + - the :ref:`Virtio Rings` + - the :ref:`Resource Table` + (optional if the RPMsg host is not the Linux kernel) + +- Optional, but recommended, mailboxes acting as a ring bell interrupt on processors to + notify of message reception + +Memory and interrupt assignments are critical design choices for any port. For a broader overview, refer to +:doc:`../protocol_details/system_considerations`. + + +Shared Memory +============= + +Shared memory forms the :ref:`physical layer` for +:doc:`RPMsg <../docs/rpmsg_design>` protocol. +The specific memory type and layout are implementation dependent, but should be a dedicated +SRAM or DDR region accessible by both cores. Caching is enabled in OpenAMP with the +`WITH_DCACHE `_ +cmake configuration. + +Memory requirements are generally modest because :doc:`RPMsg <../docs/rpmsg_design>` is a +control‑oriented protocol rather than a high‑bandwidth streaming channel. For example, using +the Linux RPMsg packet size of 512 bytes, a 8kB shared memory region can hold roughly 16 +messages. The size and message count should be tuned based on system needs. + +In addition to the messages, the Virtio ring structures need memory allocated. For Linux +this equates to 2*4k, to align with kernel page size, but likely less for other implementations. + +If the :ref:`Resource Table` is not embedded in the remote firmware image, +additional shared memory may be required for a dynamic table. + +:ref:`Remoteproc` can also use shared memory for optional +trace buffers. + + +Memory Protection +----------------- + +Because this is shared memory, appropriate hardware memory protection should be configured +on both processors. + +Depending on the memory type, this may involve configuring the Memory Management Unit (MMU), +Memory Protection Unit (MPU), or Input-Output Memory Management Unit (IOMMU) to enforce +correct access permissions. + + +Notification +------------ + +:doc:`RPMsg <../docs/rpmsg_design>` uses :ref:`ring buffers` in shared +memory, so either processor can poll for incoming messages. However, asynchronous notification +via interrupts is recommended. + +Most heterogeneous SoCs include a built‑in inter‑core interrupt mechanism, often called a +mailbox. The sending core raises the mailbox interrupt to notify the other core that a vring has +been updated for message reception, buffer release, or both. +The core can then manage the message in normal context, for example, in a thread, or in +interrupt. + + +*************** +Porting Options +*************** + +OpenAMP consists of two major components: :ref:`Remoteproc` +and :doc:`RPMsg <../docs/rpmsg_design>`. These can be ported +independently or together. + +For both there is a concept of a driver and device role which affect the virtio transport layer, +specified when setting up the virtio structure, via the definitions: + +:openamp_doc_link:`VIRTIO_DEV_DEVICE ` or +:openamp_doc_link:`VIRTIO_DEV_DRIVER ` + +When creating the virtio device (not to be confused with the device role), the driver role +indicates the main processor and the device role the remote processor. + +In OpenAMP +`RPMsg Virtio `_ +these are aliased as +:openamp_doc_link:`RPMSG_HOST ` and :openamp_doc_link:`RPMSG_REMOTE ` +respectively. + +The support for either of these roles can be enabled or disabled through the main CMake options, +as defined in OpenAMP's +`cmake/options.cmake `_ + +WITH_VIRTIO_DRIVER, sets the VIRTIO_DRIVER_SUPPORT and +WITH_VIRTIO_DEVICE, sets the VIRTIO_DEVICE_SUPPORT definition. + +.. literalinclude:: ../open-amp/cmake/options.cmake + :language: cmake + :lines: 40-41 + +These definitions affect what is compiled into the main/driver or remote/device image and can be used +to limit code size if required. By default both are enabled. + +One can think of the implementation options as four features, three of which are part of Remoteproc. + +- Full Remoteproc as life cycle management, implemented in driver role on main processor +- Remoteproc Virtio transport layer for Virtio protocol +- The resource table, implemented as part of the full Remoteproc for main processor or + independently in device role on remote processor +- RPMsg, implementing IPC and Virtio transport layer if not implemented by Remoteproc + +The resulting porting implementation approaches include: + +- On the main processor + + - :openamp_doc_link:`Remoteproc ` as driver role to manage remote processor + life cycle + - :openamp_doc_link:`Remoteproc Virtio ` to implement the virtio driver transport layer + - :openamp_doc_link:`RPMsg ` + +- On the remote processor + + - Optional :openamp_doc_link:`Remoteproc ` as device role for the + :openamp_doc_link:`resource table management ` + - :openamp_doc_link:`Remoteproc Virtio ` device role to implement the + virtio device transport layer + - :openamp_doc_link:`RPMsg ` + +Additional details and examples are provided in the subsequent sections. + +Remoteproc Framework +==================== + +The Remoteproc framework as the name suggests is responsible for remote processor management. +For both roles this includes resource table management and/or setup of the virtio transport +and as the driver role also the remote processor lifecycle management. + +Remoteproc Driver Role +---------------------- + +In the driver role, Remoteproc includes management of the remote processor lifecycle and +resource table discovery. + +Most :ref:`OpenAMP Samples and Demos` provide examples of lifecycle management for Remoteproc +in the driver role. + +The +`load_fw `_ +is an example dedicated to demonstrating Remoteproc lifecycle management. + +Remoteproc Device Role +---------------------- + +Use of Remoteproc framework for the remote or device role is optional and when used is for +resource table management, dynamically or statically. + +Dynamic examples are: + +- `ZynqMP R5 `_ + machine implementation example. + +Management using a static resource examples are: + +- `Multi Services Remote Example `_. + +Remoteproc Virtio Framework +=========================== + +When using Remoteproc, the +`Virtio transport layer `_ +is created through it (as compared to through +`RPMsg's Virtio `_). + +Remoteproc Virtio Driver Role +----------------------------- + +For the main processor driver role, Remoteproc is used + +- to obtain and configure resource table information. +- create the main processor or driver side Virtio device + +The intent of OpenAMP Remoteproc is to remain compatible with Linux's Remoteproc. + +The +`Multi Services Example `_ +example uses Linux Remoteproc for the driver role implementation. +For details refer to :ref:`RPMsg Multi Services` Demo. + +Remoteproc Virtio Device Role +----------------------------- + +For the remote processor device role, Remoteproc is used + +- to get resource table information populated by the Virtio driver +- create remote or device side Virtio device + +The +`Multi Services `_ +Example Remote shows Remoteproc Virtio device creation using +:openamp_doc_link:`rproc_virtio_create_vdev ` +and obtaining the resource table using Zephyr's +`rsc_table_get `_ +function. + + +RPMsg Framework +=============== + +The RPMsg Framework provides the IPC which is based on Virtio. + +When using Remoteproc, the Virtio used for RPMsg is setup by it for the main processor as the +driver role and device role for the remote. + +When the implementation does not require Remoteproc, OpenAMP provides Virtio through the +`RPMsg Virtio Framework `_. + +The `Zephyr OpenAMP IPC Sample `_ +is one demonstrating the use of Virtio through RPMsg framework using static vrings as the transport layer. + +The main processor Virtio device is setup as driver role, using +:openamp_doc_link:`RPMSG_HOST ` passed to +:openamp_doc_link:`rpmsg_init_vdev `. + +Refer +`Zephyr OpenAMP IPC Sample Main `_ +for an example driver role implementation. + +The remote processor Virtio device is setup as device role, using +:openamp_doc_link:`RPMSG_REMOTE ` passed to +:openamp_doc_link:`rpmsg_init_vdev `. + +Refer +`Zephyr OpenAMP IPC Sample Remote `_ +for an example device role implementation. + + +.. _hardware-abstraction: + +******************** +Hardware Abstraction +******************** + The `OpenAMP Framework `_ uses `libmetal `_ to provide abstractions that allows for porting of the OpenAMP Framework to various software environments (operating systems and bare metal @@ -107,13 +354,16 @@ compiler to GNU gcc, you may need to implement the atomic operations defined in .. _port-remoteproc-driver: + *********************************** Platform Specific Remoteproc Driver *********************************** -An OpenAMP port could need a platform specific remoteproc driver to use remoteproc -life cycle management (LCM) APIs. The remoteproc driver platform specific functions are defined -in `lib/include/openamp/remoteproc.h `_ and provided through the :openamp_doc_link:`remoteproc_ops data structure `. +An OpenAMP port could need a platform specific :ref:`Remoteproc` +driver to use :ref:`Remoteproc` life cycle management (LCM) APIs. +The :ref:`Remoteproc` driver platform specific functions are defined in +`lib/include/openamp/remoteproc.h `_ +and provided through the :openamp_doc_link:`remoteproc_ops data structure `. The remoteproc LCM APIs use these platform specific implementation of init, remove, mmap, handle_rsc, config, start, stop, shutdown and notify. These functions are passed to remoteproc @@ -127,243 +377,60 @@ by the other APIs. .. _port-remoteproc: -********************************************************************** -Platform Specific Porting to Use Remoteproc to Manage Remote Processor -********************************************************************** - -With the platform specific :ref:`remoteproc driver functions` -implemented by the port, the user can use remoteproc APIs to run application on a remote processor. - -.. doxygenfunction:: remoteproc_init - :project: openamp_doc_embed - -.. doxygenfunction:: remoteproc_remove - -.. doxygenfunction:: remoteproc_mmap -.. doxygenfunction:: remoteproc_config +Use Remoteproc to Manage Remote Processor +========================================= -.. doxygenfunction:: remoteproc_start +With the :ref:`remoteproc driver functions` required +by the framework ported, the user can call the :ref:`Remoteproc` +APIs to run an application on a remote processor, as described in the +:ref:`Remote User APIs` section of the Remoteproc design. -.. doxygenfunction:: remoteproc_stop +The following code snippets from the +`Load FW System Reference Example `_ +demonstrate the use of the Remote User APIs. -.. doxygenfunction:: remoteproc_shutdown +Remoteproc Init +--------------- +From `platform_info.c `_ -The following code snippet is an example execution. +.. literalinclude:: ../openamp-system-reference/examples/legacy_apps/examples/load_fw/platform_info.c + :language: c + :lines: 16-31 -.. code-block:: c +Lifecycle APIs +-------------- - #include +From `load_fw.c `_ - /* User defined remoteproc operations */ - extern struct remoteproc_ops rproc_ops; +.. literalinclude:: ../openamp-system-reference/examples/legacy_apps/examples/load_fw/load_fw.c + :language: c + :lines: 21-57 - /* User defined image store operations, such as open the image file, read - * image from storage, and close the image file. - */ - extern struct image_store_ops img_store_ops; - /* Pointer to keep the image store information. It will be passed to user - * defined image store operations by the remoteproc loading application - * function. Its structure is defined by user. - */ - void *img_store_info; - - struct remoteproc rproc; - - void main(void) - { - /* Instantiate the remoteproc instance */ - remoteproc_init(&rproc, &rproc_ops, &private_data); - - /* Optional, required, if user needs to configure the remote before - * loading applications. - */ - remoteproc_config(&rproc, &platform_config); - - /* Load Application. It only supports ELF for now. */ - remoteproc_load(&rproc, img_path, img_store_info, &img_store_ops, NULL); - - /* Start the processor to run the application. */ - remoteproc_start(&rproc); - - /* ... */ - - /* Optional. Stop the processor, but the processor is not powered - * down. - */ - remoteproc_stop(&rproc); +.. _port-rpmsg: - /* Shutdown the processor. The processor is supposed to be powered - * down. - */ - remoteproc_shutdown(&rproc); +*********************** +Platform Specific RPMsg +*********************** - /* Destroy the remoteproc instance */ - remoteproc_remove(&rproc); - } +In OpenAMP, :doc:`RPMsg <../docs/rpmsg_design>` uses +`VirtIO `_ to manage shared buffers. +The OpenAMP library provides a +`Remoteproc VirtIO `_ +backend implementation, and a +`VirtIO and RPMsg `_ +implementation. -.. _port-rpmsg: +You can also implement your own VirtIO backend using the +`OpenAMP VirtIO `_ and +`RPMsg `_ components provided by OpenAMP. -************************************** -Platform Specific Porting to Use RPMsg -************************************** +If you choose to create your own backend, you can use the +`Remoteproc VirtIO `_ +backend as a reference. -RPMsg in OpenAMP implementation uses `VirtIO `_ -to manage the shared buffers. OpenAMP library provides -`remoteproc VirtIO backend implementation `_. -You don't have to use remoteproc backend. You can implement your VirtIO backend with the VirtIO -and RPMsg implementation in OpenAMP. If you want to implement your own VirtIO backend, you can -refer to the -`remoteproc VirtIO backend implementation < https://github.com/OpenAMP/open-amp/blob/main/lib/remoteproc/remoteproc_virtio.c>`_. - -Here are the steps to use OpenAMP for RPMsg communication: - - -.. code-block:: c - - #include - #include - #include - - /* User defined remoteproc operations for communication */ - sturct remoteproc rproc_ops = { - .init = local_rproc_init; - .mmap = local_rproc_mmap; - .notify = local_rproc_notify; - .remove = local_rproc_remove; - }; - - /* Remoteproc instance. If you don't use Remoteproc VirtIO backend, - * you don't need to define the remoteproc instance. - */ - struct remoteproc rproc; - - /* RPMsg VirtIO device instance. */ - struct rpmsg_virtio_device rpmsg_vdev; - - /* RPMsg device */ - struct rpmsg_device *rpmsg_dev; - - /* Resource Table. Resource table is used by remoteproc to describe - * the shared resources such as vdev(VirtIO device) and other shared memory. - * Resource table resources definition is in the remoteproc.h. - * Examples of the resource table can be found in the OpenAMP repo: - * - apps/machine/zynqmp/rsc_table.c - * - apps/machine/zynqmp_r5/rsc_table.c - * - apps/machine/zynq7/rsc_table.c - */ - void *rsc_table = &resource_table; - - /* Size of the resource table */ - int rsc_size = sizeof(resource_table); - - /* Shared memory metal I/O region. It will be used by OpenAMP library - * to access the memory. You can have more than one shared memory regions - * in your application. - */ - struct metal_io_region *shm_io; - - /* VirtIO device */ - struct virtio_device *vdev; - - /* RPMsg shared buffers pool */ - struct rpmsg_virtio_shm_pool shpool; - - /* Shared buffers */ - void *shbuf; - - /* RPMsg endpoint */ - struct rpmsg_endpoint ept; - - /* User defined RPMsg name service callback. This callback is called - * when there is no registered RPMsg endpoint is found for this name - * service. User can create RPMsg endpoint in this callback. */ - void ns_bind_cb(struct rpmsg_device *rdev, const char *name, uint32_t dest); - - /* User defined RPMsg endpoint received message callback */ - void rpmsg_ept_cb(struct rpmsg_endpoint *ept, void *data, size_t len, - uint32_t src, void *priv); - - /* User defined RPMsg name service unbind request callback */ - void ns_unbind_cb(struct rpmsg_device *rdev, const char *name, uint32_t dest); - - void main(void) - { - /* Instantiate remoteproc instance */ - remoteproc_init(&rproc, &rproc_ops); - - /* Mmap shared memories so that they can be used */ - remoteproc_mmap(&rproc, &physical_address, NULL, size, - , &shm_io); - - /* Parse resource table to remoteproc */ - remoteproc_set_rsc_table(&rproc, rsc_table, rsc_size); - - /* Create VirtIO device from remoteproc. - * VirtIO device main controller will initiate the VirtIO rings, and assign - * shared buffers. If you running the application as VirtIO device, you - * set the role as VIRTIO_DEV_DEVICE. - * If you don't use remoteproc, you will need to define your own VirtIO - * device. - */ - vdev = remoteproc_create_virtio(&rproc, 0, VIRTIO_DEV_DRIVER, NULL); - - /* This step is only required if you are VirtIO device main controller. - * Initialize the shared buffers pool. - */ - shbuf = metal_io_phys_to_virt(shm_io, SHARED_BUF_PA); - rpmsg_virtio_init_shm_pool(&shpool, shbuf, SHARED_BUFF_SIZE); - - /* Initialize RPMsg VirtIO device with the VirtIO device */ - /* If it is VirtIO device, it will not return until the main - * controller side sets the VirtIO device DRIVER OK status bit. - */ - rpmsg_init_vdev(&rpmsg_vdev, vdev, ns_bind_cb, io, shm_io, &shpool); - - /* Get RPMsg device from RPMsg VirtIO device */ - rpmsg_dev = rpmsg_virtio_get_rpmsg_device(&rpmsg_vdev); - - /* Create RPMsg endpoint. */ - rpmsg_create_ept(&ept, rdev, RPMSG_SERVICE_NAME, RPMSG_ADDR_ANY, - rpmsg_ept_cb, ns_unbind_cb); - - /* If it is VirtIO device main controller, it sends the first message */ - while (!is_rpmsg_ept_read(&ept)) { - /* check if the endpoint has binded. - * If not, wait for notification. If local endpoint hasn't - * been bound with the remote endpoint, it will fail to - * send the message to the remote. - */ - /* If you prefer to use interrupt, you can wait for - * interrupt here, and call the VirtIO notified function - * in the interrupt handling task. - */ - rproc_virtio_notified(vdev, RSC_NOTIFY_ID_ANY); - } - /* Send RPMsg */ - rpmsg_send(&ept, data, size); - - do { - /* If you prefer to use interrupt, you can wait for - * interrupt here, and call the VirtIO notified function - * in the interrupt handling task. - * If vdev is notified, the endpoint callback will be - * called. - */ - rproc_virtio_notified(vdev, RSC_NOTIFY_ID_ANY); - } while(!ns_unbind_cb_is_called && !user_decided_to_end_communication); - - /* End of communication, destroy the endpoint */ - rpmsg_destroy_ept(&ept); - - rpmsg_deinit_vdev(&rpmsg_vdev); - - remoteproc_remove_virtio(&rproc, vdev); - - remoteproc_remove(&rproc); - } - -. \ No newline at end of file +For an example of setting up Remoteproc and Virtio for a remote device refer to +`zynqmp platform_info.c `_ diff --git a/openamp/glossary.rst b/openamp/glossary.rst index 02b03bc..189f1d5 100644 --- a/openamp/glossary.rst +++ b/openamp/glossary.rst @@ -8,12 +8,20 @@ Glossary AMP, `Asymmentric Multiprocessing `_ API, Application Interface + DDR (RAM), Double Data Rate (Random Access Memory) GPL, `GNU General Public License `_ + HAL, Hardware Abstraction Layer + IOMMU, Input-Output Memory Management Unit IPC, `Inter-Processor Communications `_ + IPCC, Inter-Processor Communication Controller LCM, Life-Cycle Management + MMU, Memory Management Unit + MPU, Memory Protection Unit + RAM, Random Access Memory RPC, :ref:`Remote Procedure Calls (RPC)` RTOS, Real Time Operating System RPMsg, `Remote Processor Messaging `_ SMP, `Symmetric Multiprocessing (SMP) `_ SoC, System on Chip + SRAM, Static RAM Virtio, Virtual Input Output diff --git a/protocol_details/resource_tbl.rst b/protocol_details/resource_tbl.rst index 6861b98..14daa84 100644 --- a/protocol_details/resource_tbl.rst +++ b/protocol_details/resource_tbl.rst @@ -14,6 +14,7 @@ at the offset location of each entry. Directly following the header are the reso themselves, each of which has a 32 bit type. These in remote context will likely be memory carveouts for locations of parts of the remote system and virtio device definitions. + +--------------+--------------------------------------------------------------------------------+ | Item | Description | +==============+================================================================================+ @@ -30,14 +31,22 @@ for locations of parts of the remote system and virtio device definitions. | | can correctly extract the information. | +--------------+--------------------------------------------------------------------------------+ -The fw_resource_type's are listed in the -`remoteproc header `_. +The resource table is effectively a list of resource definitions, with each entry detailed by the +corresponding :ref:`resource structure`, and the Remoteproc framework +is responsible for configuring the resources need by the remote processor (shared memory, trace +buffers, vendor specific resources). + +The possible resource types are defined in the +`Remoteproc Header `_ +and detailed in the :ref:`resource-types` section below. + Compatibility with Linux kernel ------------------------------- -The resource table should maintain compatibility with that of -`remoteproc `_. +With Linux being a key full stack operating system in the embedded devices space, the resource +table should maintain compatibility with that of +`Linux Remoteproc `_. :: @@ -45,3 +54,42 @@ The resource table should maintain compatibility with that of Related documentation can be found under the Linux kernel's remoteproc documentation: `Binary Firmware Structure `_. + + +.. _resource-types: + +Resource Types +************** + +Each of the resources in the :ref:`resource-table` is defined by a type (fw_resource_type) and +its corresponding :ref:`structure`. + +.. doxygenenum:: fw_resource_type + :project: openamp_doc_embed + + +.. _resource-structure: + +Resource Structure Definitions +------------------------------ + +The following structures correspond to each of the :ref:`resource-types` (fw_resource_type). + +.. doxygenstruct:: fw_rsc_carveout + :project: openamp_doc_embed + +.. doxygenstruct:: fw_rsc_devmem + :project: openamp_doc_embed + +.. doxygenstruct:: fw_rsc_trace + :project: openamp_doc_embed + +.. doxygenstruct:: fw_rsc_vdev + :project: openamp_doc_embed + + +In addition to the predefined :ref:`resource-types` vendors can define their +own with types between RSC_VENDOR_START and RSC_VENDOR_END. + +.. doxygenstruct:: fw_rsc_vendor + :project: openamp_doc_embed