How to Solve the “Dependency Hell” with Python

If you are programming, you know the pain of installing 3rd-party libraries or updating them. Especially if you are working on different platforms. After manual installation of a library, often you find out that a dependency (yet another library used by the one you want to install) need to be installed as well.

Dependency hell

There is another dimension to this problem – different platforms have different package systems. To name a few: Mac OSX uses MacPorts or brew, Ubuntu uses apt and CentOS uses yum. If you are like me, then you might prefer to use Mac OSX at home and one of the Linux distributions on a computational cluster. All those systems/distributions have their own libraries with different versions, compiler versions, versions of dependencies, etc.

Even if you get libraries you need, they might have different versions on different platforms. Because distributions ship their own versions. There is an official term called dependency hell.

How nice it would be to have it done automatically? No more configure scripts! Huge time saver!

The idea is to have the same set of libraries across all (Unix-like) systems. Ideally, you want to compile your software on different platforms and you want to have the same version of the required libraries across different platforms.

What libraries are we talking about? If you are programming, then there is good chance you want to install CMake, gcc, Qt, you name it.

Solution

We can break down the installation of a library into 6 actions for each platform that you are using:

1. Install dependencies (that have to be taken into account),
2. download package on each platform,
3. patch (if necessary, apply small compatibility changes),
4. configure,
5. compile the library,
6. install.

There are two ways to implement the solution:

1. Most obvious approach: do every action manually.
2. Smart aproach: automate actions with a magic Python script. One program – “dependency hell” problem solved.

Magic Python script(s)

To make use of the smart approach, meaning to automate the installation process, we suggest to write a Python script. Actually, two Python scripts: one will contain the description of the classes (we will discuss it below) and the second one will apply these classes to each library you need to install.

Python script #1

Let’s start with the first Python script that contains a base class called ExternalLibrary().

class ExternalLibrary(object):
    def __init__(self):
    name=self.GetName()
    all_libraries[name]=self
    self.environment_file_=""

The most important function in this class is the function Run() which executed the steps 1-6 from the previous section. Note that this function is recursive, because in the step 1 “Installing dependencies” also calls the function Run() for each dependency.

The code snippet below shows the function Run() and the steps 1-6 to install a library.

def InstallDependencies(self):
		# Step 1
		dependencies=self.GetDependencies()
		for dependency in dependencies:
			external_library=all_libraries[dependency]
			external_library.Run()
	
def Download(self):
		# Step 2
		if self.IsDownloaded():
			print self.GetArchiveFilename(),"already downloaded"
			return
		filename=os.path.join( self.GetCommonDownloadDirectory(),self.GetArchiveFilename() )
		url = self.GetArchiveURL()
		if type(url) is list:
			url = url[0]
		self.DownloadFile( url, filename )

def ApplyPatch(self,source_dir,patch_directory):
		# Step 3
		pass

def ConfigureBuild(self):
		# Step 4
		# This function will be overwritten depending on the configuration system
		build_directory=self.GetPackageBuildDirectory()
		self.CreateDirectory(build_directory)
		
		dir_cmd="cd {0}" .format( build_directory )
		print "dir_cmd=",dir_cmd

		configure_command="{0} {1}".format(self.GetConfigureCommand(), self.GetConfigureFlags() )

		cmd="{0} && {1} && {2}".format( self.GetEnvironmentCommand(), dir_cmd, configure_command )
		print "configure command chain=",cmd
		t=os.system(cmd)
		if t!=0:
			raise RuntimeError("Unable to configure")

def Compile(self):
		# Step 5
		dir_cmd="{0} && cd {1}" .format( self.GetEnvironmentCommand(), self.GetPackageBuildDirectory() )
		print "dir_cmd=",dir_cmd

		print "#cores=",self.GetNumberOfCores()
		make_command = "{0} && make -j {1} {2}".format( dir_cmd, self.GetNumberOfCores(), self.GetCCCommand() )
		print "make_command=",make_command	
		t=os.system(make_command)
		if t!=0:
			raise RuntimeError("Unable to make")


def Install(self):
		# Step 6
		dir_cmd="{0} && cd {1}" .format( self.GetEnvironmentCommand(), self.GetPackageBuildDirectory() )
		make_command = "{0} && make install".format(dir_cmd)
		print "make install command=",make_command
		t=os.system(make_command)
		if t!=0:
			raise RuntimeError("Unable to do make install")

def Run(self):
		# Preparations
		self.environment_file_=self.CreateEnvironmentFile()
		# Create directories
		self.CreateDirectory( self.GetCommonDownloadDirectory() )
		self.CreateDirectory( self.GetCommonSourcesDirectory() )
		self.CreateDirectory( self.GetCommonInstallDirectory() )

		# Step 1 - Install dependencies 
		self.InstallDependencies()
		name=self.GetName()
		# Go to the install directory and check whether the library is already installed
		install_dir=self.GetPackageInstallDirectory()
		self.CreateDirectory( install_dir )
		if self.IsInstalled():
			print name,"is already installed"
			return
		print name,"is not installed"
 
		# Step 2 - Download package 
		self.Download()
		self.ExtractSource()
                 
		# Step 3 - Apply patch is needed
		self.ApplyPatch(self.GetPackageSourceDirectory(), self.GetPatchDirectory() )
		self.CreateDirectory( self.GetPackageBuildDirectory() )

		# Step 4 - Build configuration 
		self.ConfigureBuild()

		# Step 5 - Compile
		try:
			self.Compile()
		except:
			print "Compile failed, try again..."
			self.Compile()

		# Step 6 - Install
		self.Install()

Let’s us have a look closer at the step number 4: configure. Configuration is done by build systems. Build systems create a make file to compile the library self and its dependencies. Different libraries use different build systems. To name a few: Qt and gcc use gnuconf, CMake and MySQL use CMake on Linux platforms. To make a distinction between different build systems, we have created two derived classes: ExternalLibraryCMake() and ExternalLibraryGNUConf(). The only difference between these two is the Configure() function.

Python script #2

The second script is a list of the libraries you need with specific parameters. For example, let’s consider installing CMake 3.4.3. Of course, we have to know

• where to download the library,
• setup the source directory,
• configuration parameters,
• dependencies.

Then the Python script will look like this:

#!/usr/bin/python

from ExternalLibrary import *

import os

class ExternalLibraryCMake_3_4_3(ExternalLibrary):
	def GetName(self):
		return "CMake343"

	def GetArchiveURL(self):
		if self.LocalDownload():
			url="http://10.0.0.17/Downloads/cmake-3.4.3.tar.gz"
		else:
			url="https://cmake.org/files/v3.4/cmake-3.4.3.tar.gz"
		return url

	def GetPackageSourceDirectory(self):
		source_dir=os.path.join( self.GetUnpackDirectory(),"cmake-3.4.3" )
		print "source_dir=",source_dir
		return source_dir

	def GetFileForTestingInstallation(self):
		return os.path.join( os.path.join( self.GetPackageInstallDirectory(), "bin" ,"cmake" ) )

	def ConfigureBuild(self):
		dir_cmd="cd {0}" .format( self.GetPackageSourceDirectory() )
		print "dir_cmd=",dir_cmd

		configure_command="./configure --prefix={0}" .format( self.GetPackageInstallDirectory() )

		cmd="{0} && {1} && {2}".format( self.GetEnvironmentCommand(), dir_cmd, configure_command )
		t=os.system(cmd)
		if t!=0:
			raise RuntimeError("Unable to configure")

As you can see, there are some checks to avoid doing the same operation twice. I.e, the package has already been downloaded or extracted.

A recursive function can easily produce the dependency graph. This graph is a visual check that all dependencies are correctly set-up:

part of the Dependency graph for our in-house code

How to handle updates?

A package update can easily be added by adding another small class that depends on the earlier version of the package. Only a couple of functions need to be changed. Once a more recent version become available, for example to install Qt 5.7.0, we have added a simple derived class ExternalLibraryQt570(ExternalLibraryQt560) based on previous library version Qt 5.6.0. The code will look like this:

class ExternalLibraryQt570(ExternalLibraryQt560):
	def GetName(self):
		return "Qt570"

	def GetArchiveURL(self):
		if self.LocalDownload():
			url="http://10.0.0.17/Downloads/qt-everywhere-opensource-src-5.7.0.tar.gz"
		else:
			url="http://download.qt.io/official_releases/qt/5.7/5.7.0/single/qt-everywhere-opensource-src-5.7.0.tar.gz"
		return url

	def GetPackageSourceDirectory(self):
			return os.path.join( self.GetUnpackDirectory(),"qt-everywhere-opensource-src-5.7.0" )

Parallel compilation

A side effect of using a Python script for automatic installation of 3rd-party libraries is parallel compilation. At the runtime, your Python script can check how many processors are available on your computer. Then your Python script automatically launches a parallel compilation and utilises all CPU power you’ve got.

Summary

Let us summarise the advantages of using the Python scripts to automate the installation of the 3rd-party libraries and solve dependency hell:

• no manual repetitive work: ones setup in the Python script, it will run automatically on different platforms;
• updates are easily added;
• all knowledge about configuration system for each library is in Python script and not lost;
• launch compilation in parallel automatically utilising multi-core CPUs.

To quote Elsa from the Disney movie Frozen: “Recursivity never bothered me anyway” 🙂

How do you update 3rd-party libraries?

Let us know in the comment box below if you would like to have the code in GitHub.

Get EZNumeric’s future articles in your inbox:

[mc4wp_form id=”916″]