Instructions

Data delivery to the SDC

  1. Request a user account. If this is the first data delivery to the SDC, email producers@sdc.upm.es to request your credentials (user name and password). If several team members are going to send data, we recommend registering with a generic team email address (not a personal one).

  2. Prepare your data for delivery, according to the following folder structure:

    • <investigation-name>
      • /documentation
      • /ancillary
      • /example
        • /data
          • /data-product-A        ← one subfolder per data product, containing the data files
          • /data-product-B
        • /metadata
          • /data-product-A        ← same subfolders as data, containing the metadata files
          • /data-product-B
      • /pre-flight
        • /data
          • /data-product-C
        • /metadata
          • /data-product-C
      • /flight
        • /data
          • /data-product-A
          • /data-product-B
        • /metadata
          • /data-product-A
          • /data-product-B
      • /postflight
        • /data
        • /metadata

    Please note:

    • This folder structure (not including data products) will be already created on the SDC data delivery area.
    • In the case of a multi-experiment investigation, there will be subfolders <experiment-name> under the investigation folder.
    • Organize the files per data products: create a subfolder per data product defined in the Investigation Data Blank Book, whenever possible.
    • The ‘metadata’ folder shall have the same subfolders as its data folder, with one metadata file for each data file.
    • Ancillary data: non-scientifically relevant data should be transferred to the ancillary folder accompanied with the list of checksums for each file. More info on the metadata and ancillary here (link)
    • Compression (optional): to speed up the data transfer, you can compress the content of each dataset folder into ZIP files.

    Sensitive data especial considerations:

    • Organize the sensitive data files by subject (one folder per subject) inside the data folder of each dataset. Encrypt the whole data folder with the SDC PGP public key* into a single encrypted file (do not encrypt each individual file separately). Do not encrypt metadata files. Transfer them unencrypted to the corresponding metadata folder.

      • <investigation-name>
        • …
        • /flight
          • /data        ← encrypt the whole data folder into one .pgp file whenever possible. Otherwise, encrypt each subject subfolder
            • /subject-1
              • /data-product-A
              • /data-product-B
            • /subject-2
              • /data-product-A
              • /data-product-C
          • /metadata        ← no need to encrypt, transfer as is
            • /subject-1
              • /data-product-A
              • /data-product-B
            • /subject-2
              • /data-product-A
              • /data-product-C
    • There are many PGP encryption solutions available, such as Gpg4win for Windows and GnuPG for GNU/Linux/BSD. If you need support on how to encrypt files using PGP, please do not hesitate to contact us.

  3. Data transfer. We offer several ways to transfer your data. Log in and detailed instructions can be found at the bottom of the page.

    • SFTP access (Recommended): Secure and efficient large-scale data transfer.
    • Web interface (Nextcloud): An alternative for users unable to use SFTP (e.g., due to firewall restrictions) or experiencing poor SFTP performance. Not recommended for large bulk data transfers.
    • WebDAV: A viable option for users facing firewall restrictions or SFTP performance issues. Unlike the Nextcloud, it has no issues on large data transfers.
    • Physical disk shipment: only in exceptional cases where electronic transfer is impractical (e.g. extremely large datasets or other technical constraints) . Prior agreement with the SDC team is required. Before sending the disk, you must provide an fsck log to verify file system integrity and a S.M.A.R.T. log to check the disk’s health status. Please contact the SDC team in advance for further details.

    4. Reminder: sensitive data must be encrypted before uploading or shipping.

  4. Data delivery notification. Once all data and metadata files have been transferred, email the SDC investigation coordinator(s) or to science@sdc.upm.es informing the investigation and dataset(s) delivered.

  5. Data verification report. SDC checks the received data and generates a report with the following information:
    • Metadata files structure check
    • Data files integrity check
    • Metadata and data files availability (one metadata file shall be found per every data file and vice versa)
    • Wrong metadata files (with wrong fields, that do not validate against the scheme)
    • Antivirus and empty files
    • Number of ingested files compared to files defined in the BB
    • Analysis of file naming and content compared to the definition in BB (only for the example dataset)

    The report will be sent to the same address from which we received the data delivery notification. If all checks pass, the delivery can be considered finished. Otherwise, we’ll request to resend the corrupt/missing data or metadata files.

* Fingerprint: 122C5A0E21B64D46BBF25A63967B50390E058ACE.

SFTP data transfer

After receiving your credentials by email, you can upload files to the SDC data delivery area via SFTP (SSH File Transfer Protocol). From the many SFTP clients available, we recommend using Cyberduck for Windows systems and LFTP for Linux systems.

The configuration parameters for our SFTP server are:

  • Host: sftp://ascella.sdc.upm.es
  • Username:<your username>
  • Password:<your password>
  • Port: 2222

To connect using the Cyberduck client, click on the Open connection button. Once the Open connection dialogue appears, insert the configuration parameters as follows:

openconnect-dialog

Click on the connect button. A notification window about the fingerprint will appear, before clicking on Allow you should check if the fingerprint matches 2d:96:c8:57:d3:08:ef:ed:c4:fb:aa:8c:06:78:eb:03. You should now be able to view the directories and files in your delivery area.

Nextcloud data transfer

Open the following URL on your web browser: https://ascella.sdc.upm.es/ and follow the on-screen instructions to log in and upload files.

You can upload files by clicking on the icon marked with a + sign or by dragging and dropping them directly into the window.

Do not overstrain your web browser by trying to upload a very large number of files (more than a few thousands) at once; it could lead to a crash under the heavy load. If you have lots of files, try to transfer them in a few batches or perhaps in a ZIP file.

WebDAV data transfer

The Nextcloud service offers the possibility to transfer files using WebDAV protocol (instead of using the web browser). Follow the instructions below to configure a WebDAV client (we recommend using Cyberduck ):

  1. Open Cyberduck and click on the Open Connection button.

  2. In the drop-down menu, select WebDAV (HTTPS) protocol.

  3. In the Open Connection window, fill in the following fields:

    • Server: ascella.sdc.upm.es
    • Port: 443
    • Username:<your username>
    • Password:<your password>

    • Click on More Options and in the Path field, enter the path of your Nextcloud directory.
      To find the path, log in to Nexcloud (see instructions above) and click on Files Settings in the bottom left corner. The path should appear – copy it starting from the “remote” part (e.g “remote.php/dav/files/your-username/”).
      Should look like this:

      webdav-tab

  4. Click on Connect.

If everything works, you should be able to see the directories of your investigations.