Exercise 2: Processing parameters in requests and processors

Task

• Create processor script that reads parameters from a velocity template to specify the bands to be combined into an RGB
• Create a request and generate a false-colour image from B8, B4, B3
• Inspect output

Material

• script s2-irg-process
• parameter velocity template s2-irg-parameters.vm
• request template s2-irg-request.json
• see /home/martin/training/parameters/ on ehproduction02

Step 1: Additional processor

There can be several processors in a processor package. Copy the additional script and the velocity template to the processor package from the previous exercise on HDFS, i.e. to

/calvalus/home/<username>/software/gdal-script-1.0

e.g.

/calvalus/home/martin/software/gdal-script-1.0

Additional explanation: At runtime on a compute node, Calvalus selects all files from the processor package that share the processor name prefix and copies them into the working directory. While copying, Calvalus strips off the prefix. s2-irg-process will become just process, s2-irg-parameters.vm will become parameters while in addition inserting parameter values.

# look into the velocity template and into the script to understand what it will do
less /home/martin/training/parameters/s2-irg-parameters.vm
less /home/martin/training/parameters/s2-irg-process
hdfs dfs -put -f /home/martin/training/parameters/s2-irg-parameters.vm /calvalus/home/<username>/software/<processor-package-name>-1.0/
hdfs dfs -put -f /home/martin/training/parameters/s2-irg-process /calvalus/home/<username>/software/<processor-package-name>-1.0/
ls -l /calvalus/home/<username>/software/<processor-package-name>-1.0

Step 2: Request with processor parameters

Write and submit a processing request.

Copy the template into the special-requests directory of your instance.

cd ~/training3-inst
cp /home/martin/training/parameters/s2-irg-request.json special-requests/
# edit special-requests/s2-irg-request.json

{
    "productionType"    : "processing",
    "productionName"    : "",

    "inputPath"         : "/calvalus/eodata/S2_L1C/v5/${yyyy}/${MM}/${dd}/S2.*_T34VFL_.*.zip",
    "dateRanges"        : "[2024-06-03:2024-06-03]",

    "processorName"     : "s2-irg",
    "processorParameters": {
        "parameters": {
            "red": "",
            "green": "",
            "blue": ""
        }
    },

    "outputDir"         : "",

    "queue"             : "general",
    "attempts"          : "1",
    "failurePercent"    : "0",
    "timeout"           : "1200",
    "executableMemory"  : "4096",

    "processorBundles"  : "",
    "calvalus"          : "calvalus-2.26",
    "snap"              : "snap-9.3cv"
}

• Insert some name of your job in the Hadoop queue into productionName, e.g. Parameter test <username>
• Insert three band names that have the same resolution in Sentinel-2 MSI L1C into the values of red, green, blue, e.g. B08, B04, B03
• Insert an output dir starting with /calvalus/home/<username>/ into outputDir, e.g. /calvalus/home/martin/parameter-test
• Insert the path to your processor bundle into processorBundles, i.e. /calvalus/home/<username>/software/gdal-script-1.0

Step 4: Request submission

Submit your request.

If you have logged in again after the previous exercise

. mytraining3

Then, submit the request with the Calvalus Hadoop Tool cht. You may use -a with cht to submit the requst asynchronously without waiting for it to finish.

cht -a special-requests/s2-irg-request.json

You can monitor status using the job id reported by submission with

cht --status job_xxxxx_yyy

In case of failure you can use the same commands as listed in exercise 1:

yarn application -list -appStates FAILED | grep <username>
yarn logs -applicationId application_<nnnnn>_<mmm> -log_files stderr,stdout | less

Try to find out what is wrong, correct it, and re-submit your request. Please, ask if you do not succeed.

Step 5: Result inspection

Download the result (e.g. with filezilla) and open it in a viewer, e.g. in QGIS. Adjust the colour scale (e.g. to values between 300 and 5000 per band) to make visible the ground.