![]() Copy the following test code and put it in a test file in this folderįrom pathlib import Path from pyspark.sql import SparkSession from spark_pipeline_framework_testing.test_runner import SparkPipelineFrameworkTestRunner def test_folder ( spark_session : SparkSession ) -> None : data_dir : Path = Path ( _file_ ).(Optional) Create an output_schema folder and put in any schemas you want applied to the output views.(Optional) Create an output folder and put in files that represent the output views you expect. ![]() This follows the Spark Json Schema format. (Optionally) Create an input_schema folder and put in any schemas you want applied to the above views.Create an input folder and put in files that represent the input views.Create a folder structure similar to the folder structure of your library in SparkPipelineFramework (This is how the Testing Framework finds the Transformer to run).Testing framework that can tests SparkPipelineFramework library by just providing input files to setup before running the transformer and output files to use for verifying the output Usage ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |