This page aims to collect the comments on the Document "STAC Requirements for PRIP", downloadable from those links;
| Version | Link | Description |
|---|---|---|
| 1.0-RC1 | CAP-TN-0XX-BE_STAC_Requirements_for_PRIP-1.0-RC1.docx | Preliminary version that will be updated with detailed requirements regarding downloading of assets from S3 buckets. |
| 1.0-RC2 | CAP-TN-036-BE_STAC_Requirements_for_PRIP-1.0-RC2.docx | This version takes into accounts Jolyon’s comments on version 1.0RC1 and defines additional requirements for PRIP Instance and PRIP Client, in particular regarding the download of EOPF products from S3 buckets. |
| 1.0-RC3 (current) | CAP-TN-036-BE_STAC_ICD_for_PRIP-1.0-RC3.docx | Update considering comments from Jolyon and Mario. |
Different solutions are identified for storing and retrieving assets on S3 buckets. Those solutions are described in following presentation.
Storing ZARR on S3 for STAC.pptx
A benchmark of those solutions has been performed. Results are provided in following presentation:
Benchmark-Storing-and-downloading-EOPF-Products-on-S3-Buckets.pptx
Comments on the current version of the ICD can be added in the following table:
| Id. | Author | Section/Page number | Title | Comment | Answer to comment | Document changed | Status |
|---|---|---|---|---|---|---|---|
| 1 | DLR | 2 Objectives and needs, page 8 | Scope of document | The STAC concepts are already descibed in [STAC_EOF_ICD]. The scope of this section could be more on the PRIP and its objectives and context, as in the current OData ICD. | Indeed, the first paragraph of this section is taken from the EOF ICD. It is just to remind fundamental concept of the STAC specification. The following paragraphs in this section are really focus on the objectives of this ICD, that is specifying interface requirements for PRIP. Not clear for me what you would suggest to detail in this section. | No | Open |
| 2 | DLR | 3.2 Nominal use scenario description, page 12 | Queryables and collection metadata | There is a strong relation between the Queryables, the STAC data model and the use cases of the service. Therefore we would expect this ICD to define the exact set of Queryables and the collection metadata for the PRIP. | It has been discussed with Jolyon and we agreeded that the queryable schemas should be provided in tailored ICD or conformance test suite as per STAC-PRIP-COL-REQ-0030. | No | Closed |
| 3 | DLR | 3.2 Nominal use scenario description, page 12 | Sequence diagram | The sequence diagram indicates to query the catalog metadata, queryables and collection metadata every time before doing the product search. But systematic downloaders doing regular search requests usually know the Queryables and collection metadata (see comment above). The diagram could be streamlined to reflect this case. | You are correct. Updated schema and related explanation to reflect this. | Yes | Closed |
| 4 | DLR | 3.2 Nominal use scenario description, page 12 | Downloading zipped EOPF products | The downloading step could be extended to also reflect the download of a zipped EOPF product (zipped Zarr hierarchy). In this case, the S3 list objects operation is not needed. | As discussed with Jolyon, the PRIP S3 buckets will store ZARR in their expanded form. Therefore, to prevent double storage capacity, it is not foreseen to support downloading of ZARR as ZIP archive (note that ZARR files are compressed). | No | Closed |
| 5 | DLR | STAC-PRIP-ITEM-REQ-0050, page 21 | Product Type | Why is an explicit Product Type property needed here since the product type is already reflected by the collection, as stated in STAC-PRIP-COL-REQ-0020? | Note that this property is provided in the metadata of the ZARR file. Not having a strong coupling between this field and the collection name could be interesting for missions with large set of product sub types and for which it would not be relevant to manage all those sub types at collection level. | No | Closed |
| 6 | DLR | STAC-PRIP-ITEM-REQ-0050, page 21 | Expiration Date | The "unpublished" property would perhaps be semantically closer to the OData property EvictionDate than "expires". | From my understanding of the timestamps extension, the unpublished property is set when the item metadata are actually no longer valid. Whereas the expires field is at item creation time to inform the client when the item and related assets will no longer be available. I don't know the clear semantic of the OData EvictionDate property. I guess this is something to discuss to choose the more appropriate. | No | Closed |
| 7 | DLR | STAC-PRIP-ITEM-REQ-0090, page 24 | Local Folder Name | What is the purpose of Local Folder Name property here? In case of an asset's href pointing to an individual group of measure or band of the Product, the local folder name is already included in the S3 URL as prefix. | It is a convenient method proposed to STAC Client to name consistently the asset they download without to extract this value from the href field. | No | Closed |
| 8 | DLR | 3.1 Architecture overview, page 10 | S3 Authentication | Requests to the STAC API require authentication via a bearer token. However, both the text and the architecture diagram in Section 3.1 suggest that requests to and downloads from the S3 store currently do not require authentication. The same applies to the command in 3.4.3.4 Downloading ZARR content (see comment below). Later in the document, the requirement "STAC-PRIP-API-REQ-330 – S3 API endpoint authentication" (page 27) indicates the need for an authentication mechanism. To enhance security, an authentication mechanism should also be enforced for the S3 API. If a technical solution for such a mechanism is already planned, it should be explicitly documented and specified in CAP-TN-036-BE. Additionally, would it be feasible to use the same bearer token for both the STAC API and the S3 API to ensure consistency and simplify the implementation? | Added a note in the schema and an explanation in the related text to clarify that all requests sent to the STAC catalogue and S3 buckets shall be authenticated. From my knowledge, Cloud providers does not support bearer token to access to S3 buckets. Requirement STAC-PRIP-API-REQ-330 has been updated to specify consistent authorization between Catalog access and S3 Bucket access. | YES | Closed |
| 9 | DLR | 3.2 Nominal use scenario description, page 11 ff. | Authentication procedure | The authentication procedure for the STAC API (retrieving, submitting, and verifying the bearer token) is missing from the use scenario, even though it constitutes an essential part of the workflow. The same applies to the authentication mechanism for the S3 API. | A common ICD is foreseen to better describe how a service shall validate the bearer token and implement the expected authorization approach. | NO | Closed |
| 10 | DLR | 3.4.1.2 Authentication and authorization requirements, page 26 ff. | Automatic authentication token retrieval without manual interaction | Systematic downloaders must be able to automatically retrieve authentication tokens without requiring any manual interaction. A requirement should be added to address this need. Similarly, the authentication mechanism for the S3 API should also avoid manual interaction. In "STAC-PRIP-API-REQ-380 – Keeping an access token up-to-date," it is specified that the PRIP Client is responsible for maintaining its access token by leveraging the refresh token OIDC flow. However, the details of the refresh token OIDC flow are not specified in the document and should be elaborated further. Key questions include: How long is a refresh token valid? Can a refresh token be retrieved automatically? As stated above, manual interaction should be minimized wherever possible. | This shall be addressed by the common ICD related to the Centralized Authentication Service. Note that the Systematic Downloader use case is well supported by OIDC Client Credential Flow. | No | Closed |
| 11 | DLR | 3.4.2 PRIP Client requirements, page 28 | PRIP client authentication for S3 download | A requirement specifying the PRIP client authentication mechanism for S3 downloads, analogous to "STAC-PRIP-API-REQ-370 – PRIP Client authentication for STAC browsing," should be added to define the authentication procedure for the S3 API. Would it be feasible to use the same bearer token for both the STAC API and the S3 API to ensure consistency and simplify the implementation? | Added requirement STAC-PRIP-API-REQ-391 | YES | Closed |
| 12 | DLR | 3.4.3.4 Downloading ZARR content, page 37 | Downloading procedure | The command does not include authentication against the S3 API. However, an authentication mechanism is needed as stated in one of the comments above and in "STAC-PRIP-API-REQ-330 – S3 API endpoint authentication". | Added a note after the command to clarify that AWS CLI is assumed to be properly configured. | YES | Closed |
| 13 | DLR | 3.4.3 Use case examples, page 30 ff. | Detailed examples for product search | It would be helpful for a client to have more examples for querying product especially in terms of filtering (by name, geography, datetime), counting and paging through search results. | Added some details regarding fetching pages. For more specific and complex use cases, implementors are invited to read the STAC documentation available on line. | YES | Closed |
| 14 | DLR | General | Quotas | The current PRIP is supposed to apply quotas to orders and downloads. If this is also a requirement for the STAC-based PRIP, a corresponding section should be added to the document. | Is it really needed to implement such quotas for internal EOF services, assuming that credentials will be provided to well known and trusty clients. To be discussed | NO | Open |
Add Comment