Matillion

Of standard libraries installed. Additionally, and frequently, Matillion ETL users may want to install additional Python libraries on their Matillion ETL instances. This can be done using pip, Python's package manager. Read Additional Modules to learn how to add Python libraries to a Matillion ETL instance.As of version 1.61.6, Matillion ETL comes with version 8.1.2 of pip for Python2 and version 9.0.3 of pip for Python3. Typically, the version of pip that ships with Matillion ETL should be sufficient for most use cases and we do not recommend updating the version of pip. Matillion ETL releases are tested based on the version of pip that is included with Matillion ETL. Installing a different version of pip could result in unexpected behavior within Matillion ETL. If you find a need to upgrade the version of pip on your Matillion ETL instance, please ensure you have a current backup of your instance that can be restored if necessary.Migrating Matillion InstancesThere can be many reasons to launch a new Matillion ETL instance and migrate your existing Matillion ETL instance over to it. The most common scenario for this is when updating to a new version of Matillion ETL following our recommended best practice to launch a new Matillion instance and then use Matillion's Migrate feature. When migrating, note that Python libraries are included in the list of resources that are not automatically migrated. We will outline here the best practice steps for migrating Python libraries that your Matillion ETL jobs may have dependencies on, to a new Matillion ETL instance.Confirm versions of Python and Pip on the CURRENT Matillion ETL instance.SSH on to the Matillion instanceRun the following commands and note the output:Current version of Python2 installed: sudo python --versionCurrent version of Pip installed for Python2: sudo pip --versionCurrent version of Python3 installed: sudo python3 --versionCurrent version of Pip installed for Python3: sudo pip3 --versionConfirm the Python Libraries installed and the version of each on the CURRENT Matillion ETL instance.SSH on to the Matillion instanceRun the following commands and note the output:List of installed Python2 libraries: sudo pip freezeList of installed Python3 libraries:

Sudo pip3 freezeConfirm versions of Python and Pip on the NEW Matillion ETL instance.SSH on to the Matillion instanceRun the following commands and compare the output against the CURRENT Matillion ETL instance. The NEW Matillion ETL instance should be of the same or later version of Python2/3 as the CURRENT Matillion ETL instance.Python2 sudo python --versionPython2 sudo pip --versionPython3 sudo python3 --versionPython3 sudo pip3 --versionConfirm the Python Libraries installed and the version of each on the NEW Matillion ETL instance.SSH on to the Matillion instanceRun the following commands and note the output. Note that by default, a new Matillion instance will have some Python libraries installed.Python2 Libraries: sudo pip freezePython3 Libraries: sudo pip3 freezeCompare the list of Python Libraries on the CURRENT Matillion instance against the list of Python Libraries on the NEW Matillion instance. Based on the comparison, identify the Python libraries that need to be installed on the new instance.Install each Python library that is required on the NEW Matillion instance, ensuring to install the exact same version of the library installed on the CURRENT Matillion instance.Python2 - Install a library of a specific version:sudo pip install [modulename]==[version number]Example: sudo pip install boto3==1.14.53Python3 - Install a library of a specific version:sudo pip3 install [modulename]==[version number]Example: sudo pip3 install boto3=1.17.45Advanced TopicsThe topics below focus on "advanced topics" related to using Python with Matillion ETL. Many of the sections below link to other documentation topics.Disabling Python on a Matillion ETL instanceIt is possible to disable the Python Script feature on a Matillion instance. Following are the steps to do so:SSH on to the Matillion instanceMake a backup of a configuration file that will be edited:sudo cp /usr/share/emerald/WEB-INF/classes/Emerald.properties /usr/share/emerald/WEB-INF/classes/Emerald.properties.backupEdit the same configuration file:File: /usr/share/emerald/WEB-INF/classes/Emerald.propertiesAdd the following to the end of the file: ALLOW_PYTHON_COMPONENTS=falseRestart the Matillion service.From an SSH session:sudo service tomcat stopsudo service tomcat startFrom a Matillion UI session:Click Admin → Restart ServerRestricted UsersBy default, when a Python Script component executes, it runs as an external process on the Matillion ETL server directly, with the same privileges as the web server. It is possible to restrict the execution of the Python

Managing Python on a Matillion ETL virtual machine (VM)This guide was provided by Matillion's solution architect team.Python is a popular programming language for data engineers. Recognizing this, Matillion ETL offers a Python Script component, which allows Matillion users to integrate their own Python scripts into their Matillion ETL jobs. When using Python scripts in Matillion ETL jobs, an important consideration is if the nature of the Python is appropriate to run on a Matillion ETL instance, which is typically sized for an ELT workload. Read Executing Python Scripts Outside of Matillion for more insights into that topic.When Matillion ETL users reach out to Matillion Support related to their use of Python, we've found that, often, the issues raised relate to the management of Python on their Matillion ETL instances. The aim of this guide is to highlight some best practices around managing your Python environment on a Matillion ETL instance.Best PracticesDo not uninstall Python on a Matillion ETL instance. An alternative is to disable the availability of the Python Script component.Do not change the version of Python installed on a Matillion ETL instance.Maintain an inventory of any manually installed Python libraries, including the version of the Python library.When automating the deployment of new Matillion ETL VMs, account for any Python library dependencies as part of the automated deployment.When using the Python Script component, use the Python3 interpreter.Do not change the version of pip installed on a Matillion ETL VM unless there is a Python library dependency. If required to update the version of pip, ensure you have a current backup of the Matillion ETL VM.Python Interpreters and VersionsThe Python Script component in Matillion ETL allows a user to run a Python script against different Python interpreters: Jython, Python2, and Python3.The Jython interpreter is a Java interpreter, based on Python2, for running Python scripts within a Java application. Matillion ETL is written in Java, which is where the support for this interpreter comes from. Historically, the main benefit of using the Jython interpreter is that it allows the Python script execution to use the connection to the cloud data warehouse (CDW)