Applications (Apps) | Quakeworx Gateway

Topics

Overview
▶ Creating the Application
▶ Building Input Elements
▶ Quakeworx UI Webform Elements

Overview ↑

Applications (Apps) are software applications that are configured to be executed on a predefined compute system. Launching an App creates a Job that stores its configuration and state.To run a job on a system, you will need access to an application. An App encapsulates all the necessary details to execute a job on a system and generate results. Each App is versioned and owned by a designated user who holds special privileges over the application.

Requirements
All apps must have a singular entry point which takes all arguments.
Containers
Must not hard code host side port (use environment variable or argument)
Must not hard code volume mounts
VNC apps must include a VNC server and an option to set a password (env variable)

The App definition provides the information required by the jobs service to:

Stage input files before running the application.
Launch the application on the specified system.
Monitor the application's progress during execution.
Archive output files after the application completes.

This structure ensures that jobs can be efficiently managed and executed across systems.

Tutorial:Creating an Application ↑

This tutorial will walk you through the steps of creating an application to run on a AWS EC2 system using the provided web interface. Follow the instructions below to complete each section of the form.

Application overview ↑

The image classify application allows you to perform image classification using TensorFlow on AWS EC2.

Once you submit a job, image classify handles everything automatically. The input files you provide during the submission are staged, and the classification process is triggered. You can then find the results in the designated output directory.

To classify an image, simply pass the image file through the command line using the --image_file argument. In order to test the application using the docker image, run a command like this:

docker run --rm osp/img-classify:v1 --image_file=https://s3.amazonaws.com/cdn-origin-etr.akc.org/wp-content/uploads/2017/11/09152345/Alaskan-Malamut…

This will classify the image found at the given URL, which is a JPEG of an Alaskan Malamute. If everything works correctly, you should see the classification results as output in the job output directory.

Step 1: Accessing the application creation page ↑

Login to https://qwx1.onescienceway.com.
Navigate to the “Apps” menu, https://qwx1.onescienceway.com/node/add/tapis_app and click on "Add app".

Step 2: Set the status ↑

Under the Status section, you will find three options:

Enabled: Select this if the application should be available for use.
Disabled: Select this to disable the app.
Archived: Choose this if you want to archive the app but still retain its details for future use.

Select the appropriate status for your application. For this tutorial, choose Enabled.

Step 3: Provide app details ↑

In this section, you need to fill in the following fields:

App nickname: Enter a short nickname for the application. This is a required field.
App ID: Provide a unique ID for the app. It must be in lowercase and without any spaces. For this tutorial, use your user ID. (e.g., <your user ID>.apps.image_classification.aws).
App Version: Specify the version of the app (e.g., 0.1.0).
Summary: Add a one-line summary that describes the application.
Long Description: You can include a detailed description of your app in this section using the rich text editor.

Step 4: Select app type ↑

In this section, choose the type of application from the following options:

Batch: Select this if your app runs as a non-interactive command line application.
VNC: Choose this for interactive graphical applications like MatLab.
Web: Use this for interactive web applications like Jupyter Notebook.

For this tutorial, select the option, Batch.

Step 5: Miscellaneous settings ↑

Restartable: Check this box if the job can be restarted by the user after submission.
Show generated command: Displays the command in the message window for troubleshooting.
Update the app’s webform: If checked, this will update the app settings with the values from the form.

Step 6: Select container runtime ↑

Next, choose the runtime that will be used by the application. The available options are:

Docker
Singularity
- If this is chosen, additional runtime options like Singularity start or Singularity run become available.
Executable
- If this is chosen, specify the Executable path to point to the location where the script or binary resides.

For this tutorial, the app uses Docker containers, select Docker.

Step 7: Batch scheduler ↑

Use batch scheduler: Check this box if the app requires a batch scheduler like SLURM.
- If the app uses a batch scheduler, enable this option and add scheduler profiles.

Step 8: Container image setup (For Docker or Singularity runtime) ↑

This section is crucial for defining how your application will be packaged and deployed within a container.

Container image source:
- Public image name: Provide a publicly-accessible container image: Select this if your application's container image is readily available from a public container registry (like Docker Hub). You'll need to provide the full image name (including repository and tag) in the "Container Image URI" field below.
- Source: Upload a zipped folder with the source to build the container image: Choose this option if you have the source code and a Dockerfile to build the container image. You'll be prompted to upload a zipped archive containing these files. Currently it is not supported.
- Image: Upload a pre-built container image file: Select this if you already have a pre-built container image file (e.g., a .tar file) that you want to use directly. You'll be asked to upload this file. Currently it is not supported.
Container image URI:
- This field is where you specify the exact location or identifier of the container image.
- If you chose "Public image name" above, enter the full image name here (e.g., ubuntu:latest, tensorflow/tensorflow:2.5.0).
- If you uploaded a source code archive or a pre-built image, this field might be auto-populated or you might receive further instructions on how to reference the image.

For this tutorial, select the option, Public image, and put the image URI, osp/img-classify:v1.

Step 9: System selection ↑

Available systems: Select the system(s) where the app can be executed. For example, you can choose ”AWS system”.
Default system: Choose the default system for the application.
Tenant: Select a tenant from the dropdown.

For this tutorial, select the option for Available systems and Default system, AWS tutorial system, and select the Tenant, tutorial.

Step 10: Input Type Configuration ↑

Define how inputs will be provided to the app:

No input parameters are needed: Select this if your application doesn't require any user-provided input.
Fixed Command: Specify the full command to run the application: Choose this if you want to provide a specific command that will be executed when the application launches. You'll enter the command in the text field that appears.
Form: Create a form that will allow users to set parameters, upload files as input to the application: Select this option to create a user interface where users can input parameters or upload files when they launch the application.

For this tutorial, users can input parameters, select Form and provide the necessary configuration.

Step 11: MPI setting (Not used in this tutorial) ↑

Uses MPI: Check this box if your application uses MPI for parallel processing across multiple nodes or cores.
MPI Command: If you checked "Uses MPI", specify the MPI command that should be used to launch your application (e.g., mpirun, srun, etc.).

Step 12: Input Parameters (Environment Variables, Container Args, Scheduler)(Not used in this tutorial) ↑

In the Parameters section, you can specify various inputs required for the container:
- Environment variables: Add environment variables one per line in the key=value format. For instance: MY_APP_PORT=8080.
- Container launch arguments: Include arguments in the required format, with each argument and its value on a new line. For example: --volume /mnt/nfs:/mnt/nfs.
- Batch scheduler options: If your application uses a batch scheduler (e.g., SLURM), include any specific options here. For example: --cpus-per-task 2.

Step 13: Configure Resource Limits (Not used in this tutorial) ↑

Now, configure the resource limits for the application:

Max jobs: Set the maximum number of jobs that can run concurrently for this application.
Max jobs per user: Define the maximum number of jobs one user can run simultaneously.
Number of nodes: Specify the number of nodes your application requires for execution. If your app is parallelized, users can override this value. You can allow users to override the number of nodes.
Cores per node: Define how many CPU cores per node will be assigned to this app. You can allow users to override cores per node.
Memory: Define the amount of memory required for the app, with an option to allow users to override this value. (e.g., 100 MB).
Max runtime: Specify the maximum runtime for the app and give users the ability to override this if necessary (e.g., 10 minutes).

Step 14: Finalize and save ↑

After completing the form, review all the details to ensure correctness.
Click on the Save button to register the application.

Tutorial: Creating Input Element for Your Application Form ↑

In this tutorial, we will walk through the steps of adding custom input elements to an application form. This process is important if you want to collect specific input data from users running your scientific application on a HPC cluster or AWS system. The following instructions will guide you through the process using the web interface shown in the images.

Step-by-step guide to adding Webform elements

Step 1: Application created ↑

Once you have successfully created an application, you will see a confirmation message:
"An app form has been created. App Image classification has been created."

Click the "Configure app form" button to start setting up the custom input elements that your app will require. The app will be configured to accept user inputs necessary for running the scientific application on a designated system (such as a HPC cluster or AWS).

Step 2: Begin configuring the app form ↑

After clicking "Configure app form", you will be directed to a page where you can specify job resources and add input fields for your application. In this case, we are working with an Image classification application as an example.

Step 3: Search for elements ↑

After clicking the Build link, to add a custom input field, such as a URL for the image in this example, follow these steps:

Select the Add element button on the top left.
In the Select an element search box, type a keyword related to the input you need. For example, if you need a text field, you can search for the "text field".
From the list, choose the desired element. In this example, the Text field (OSP) element is chosen to capture the image_url.

Step 4: Add and configure the Text Field ↑

After selecting the text field element, you need to configure it to match your application requirements.

Title: Enter the title of the input field (e.g., image url). This is how users will know what to input.
Key: The key is automatically generated based on the title (e.g., image_url).
Command arguments: This section allows you to define how the input will be passed to the backend. In this case:
- Type: Select Command line argument.
- Command prefix: Set the prefix, for example, --image_file=
- Ensure the Add trailing space to command prefix checkbox is unselected so that the command is formatted correctly.
- Default value: You can provide a default value, like the URL of an example image (https://s3.amazonaws.com/cdn-origin-etr.akc.org/wp-content/uploads/2017/11/09152345/Alaskan-Malamut…).

Once everything is configured, click Save to add this element to the form.

Step 5: Review added element ↑

After saving, the newly added text field (image url) will appear in the element list on the right side of the screen. This list shows all the input fields that will be included in the final application form.

At this stage, you can continue adding more elements or adjust the current ones as needed.

Step 6: The final app form ↑

Once you’ve added all necessary input fields, the app form will be complete. In this example, after adding the text field for the image url, it is displayed in the form alongside other app settings.

You may update the following information if necessary:

Number of nodes: Set this based on how many nodes you want the application to use.
Cores per node: Specify the number of cores per node.
Memory (MB): Define the memory requirements.
Max runtime (minutes): Set to 10 minutes.
System: AWS system or cluster (AWS system in this case).
Job name: Enter a name for this job. In this example, the job name is set as Image classification.
Image URL: https://s3.amazonaws.com/cdn-origin-etr.akc.org/wp-content/uploads/2017/11/09152345/Alaskan-Malamut…

Once you’ve filled out these details, click "Submit" to proceed, in order to launch the job.

Proceed to the next document: ‘Job submission’

Tutorial: Quakeworx UI Webform Elements ↑

Overview ↑

Quakeworx introduces UI Webform elements specifically designed to handle complex input scenarios typical in scientific computing applications. These UI webform elements are built on top of Drupal Webform elements and offer advanced options to pass user inputs as command-line arguments, environment variables, or scheduler options. This flexibility makes UI webform elements highly useful for scientific applications that need to interact with command-line tools and job scheduling systems.

List of stock UI webform elements ↑

There are seven stock UI webform elements, which build on the base Webform elements in Drupal, provide additional configurations tailored to pass data as commands or environment variables. The list of Quakeworx elements includes:

Checkbox (OSP): Integrates command-line options with checkbox states, enabling conditional logic based on user selection.
Computed Twig (OSP): Dynamically generated values using Twig templating, useful for on-the-fly calculations.
File (OSP): Facilitates file uploads with direct integration into command-line arguments or environment variables.
Number (OSP): Allows for numeric input with validation and direct command integration.
Radios (OSP): Passes user-selected radio button options as part of command execution.
Select (OSP): Dropdown selections are directly passed into the command-line or scheduler.
Text field (OSP): Passes user text inputs as arguments, allowing for custom command execution.

Each of these elements is optimized to integrate user inputs into command-line workflows, making Quakeworx a powerful solution for scientific applications that require robust input handling and backend tool integration.

Common options for UI webform elements ↑

All Quakeworx elements share several advanced configuration fields to fine-tune how data is passed and processed:

Command type
- Choose how the value is passed (as a command-line argument, environment variable, or scheduler option):
  - Command-line argument: The element information is directly incorporated into the command-line arguments.
  - Environment variable: The element information is passed to the command via an environment variable.
  - Scheduler option: The element is potentially integrated with a job scheduler.
- This feature allows for better integration with command-line-based systems or job schedulers.
Command prefix/suffix: Add custom strings before or after the value for enhanced command structure (e.g., --input-prefix=, --output-suffix).
Add trailing space to prefix/suffix: Adds a space after the command prefix or suffix, ensuring that commands are well-formed when concatenated.
Ignore value: Skip the element's value entirely and use only the prefix/suffix if needed.
Ignore empty values: Skips any elements with empty values when building the command. This prevents null or unnecessary inputs from being passed to the command line.
Show only on restart: Useful for jobs that need certain fields only when restarting a task.

These configurations ensure that the input can be accurately passed to external systems or scripts in scientific workflows.

Examples of stock UI webform elements ↑

1. Checkbox (OSP) ↑

The Quakeworx Checkbox allows the state (checked/unchecked) to influence command execution. For instance, if the checkbox is checked, you can enable specific arguments in the job configuration.

Additional Quakeworx configuration options:

Ignore value: This option disregards the checkbox value (true/false) and prevents the inclusion of any prefix or suffix when constructing the command.
Ignore element key: If you specify the element key, such as "qwx1_select_test," this option allows the checkbox value to be ignored and prevents the inclusion of any prefix or suffix when generating the command.

Example usage:

If the checkbox is meant to control file uploads, you can configure it to add a prefix of --enable-upload and a suffix of --path=/data if the checkbox is checked. The full command might look like:

run_data_analysis.sh --enable-upload --path=/data

If unchecked, nothing will be added to the command line.

2. File (OSP) ↑

The Quakeworx File element handles file uploads and can pass the file path or directory as part of the command-line or environment variable.

Additional Quakeworx configuration options:

Command Value: This option allows the user to specify what part of the file information is passed. The options typically include:
- File path: The full path to the uploaded file is included.
- File directory: The directory containing the uploaded files is included.
- None: Neither path nor directory is included, but the file remains accessible in the designated input location.
Make this the last command argument: Ensures that the file path or directory appears as the last argument in the final command. This is particularly useful for command-line tools that expect the file input at the end of the command.
Ignore value: This option prevents the inclusion of any prefix or suffix when constructing the command.
Repeat command arguments:
- Replicates the command arguments for scenarios involving multiple file uploads.
- Example: If multiple files are allowed and the prefix is --input-file, the command could be:

process_data.sh --input-file a.txt --input-file b.txt

Example usage:

If a user uploads a file sample_data.csv and you could configure the element with the prefix --file=, to pass the file path, the resulting command looks like:

process_data.sh --file=sample_data.csv

3. Select (OSP) ↑

The Quakeworx Select element extends the standard dropdown with command-line integration based on the selected option.

Example usage:

If the user selects High Performance from a dropdown, and the prefix is --mode=, the resulting command might be:

run_analysis.sh –mode=HighPerformance

4. Text Field (OSP) ↑

The Quakeworx Text Field element passes user input directly into the command as a string.

Example usage:

If the user inputs science into the text field, and the prefix is --keyword=, the command would be:

run_analysis.sh --keyword=science

5. Radio (OSP) ↑

The Quakeworx Radio element allows a single selection to be passed as a command option. Each radio button corresponds to a unique value passed into the backend system.

Example usage:

If the user selects "SDSC" from a group of radio buttons, and the prefix is --computing-center=, the resulting command looks like:

run_process.sh --computing-center=SDSC

6. Number (OSP) ↑

The Quakeworx Number element allows the user to enter a numeric value that will be passed as a command-line argument, environment variable, or scheduler option.

Useful configuration options:

Minimum/Maximum Value: Defines the allowed range for the numeric input.
Step: Determines the increment by which the number can be adjusted. For example, if the step is set to 1, users can only enter whole numbers.
Default Value: Predefines a number that will be used unless the user changes it.

Example usage:

If the user enters the number 8 in a field where the prefix is --cpus=, the resulting command looks like:

run_process.sh --cpus=8

If the suffix is GB and the user enters 16, the command looks like:

run_process.sh --memory=16GB

7. Computed Twig (OSP) ↑

The Quakeworx Computed Twig element allows for more advanced logic, where the value is not directly user input, but dynamically generated using Twig templates. This value can be passed to the backend as a command-line argument, environment variable, or scheduler option, based on conditions or calculations defined in Twig.

Useful configuration option:

Twig Expression:
- This is the most critical part of the Computed Twig element. The Twig expression is used to dynamically generate a value based on other form inputs, variables, or conditions.
- Example: Using Twig to calculate a total or to format a value before passing it to the command.

Example usage:

Imagine you want to compute a total price based on the number of items a user selects in another form element and a constant price per item:

{% set price_per_item = 20 %}

{% set total = data.number_of_items * price_per_item %}

This could then be passed to the command with a prefix like --total-price= (for 5 items, each costing 20 units), generating a command like:

checkout.sh --total-price=100

Important considerations

Twig knowledge: Familiarity with Twig syntax and functions is essential for effectively utilizing this element.
Performance: Complex computations or frequent updates can impact form performance, especially with AJAX enabled. Use caution and consider alternative approaches for resource-intensive operations.

Example: Adding a Webform Element for the "Image classification" App ↑

To add a custom webform element to the "Image classification" app, follow the steps below:

Step 1: Select the "Image classification" App ↑

Navigate to the Apps menu on the left sidebar.
Under All apps, you will find a list of available applications.
Locate the Image classification app and click on it to proceed.

Step 2: Configure the Webform ↑

After selecting the "Image classification" app, the app's summary page will appear.
Click the Configure app form button to begin editing the webform for job submission.

Step 3: Access the form builder ↑

Once you click Configure app form, you will be directed to the job page.
On the job page, locate the Build link at the top and click it to access the form builder.

Step 4: Open the Webform builder ↑

After clicking the Build link, the form builder page will open.
Here, you can add, update, or delete webform elements.
Click the Add element button at the top of the page to open the element selection window.

Step 5: Add a Webform element ↑

In the Select an element pop-up window, a list of webform elements will appear.
Type “osp” in the search bar to filter relevant webform elements provided by the Quakeworx module.
Choose the desired element by clicking the Add element button next to the element’s name.

Conclusion ↑

The Quakeworx extended Webform elements provide a highly configurable, powerful way to handle inputs in scientific workflows. By allowing data to be passed as command-line arguments, environment variables, or scheduler options, Quakeworx enables seamless integration with backend command-line tools and job schedulers. This flexibility simplifies complex workflows and enhances the usability of scientific applications built on Drupal Webforms.